Yesterday I decided to swap the video cards back.
Regardless of the good cooler, the 9800GT was becoming too hot. Apparently there is just not enough air and ventilation - thanks again to Dell motherboard design. The video card is positioned at the very bottom, next to the only PCI slot, so there is less than an inch space between the radiator and the bottom of the main case chamber. The optional fans of the Turbo Module barely fit there, almost touching the wires. Under heavy load the card temperature was over 100°C!
So I decided to return the original ATI HD5670 back - it's a bit less powerful, but being newer it uses less power than 9800GT, so runs much cooler. I put the card back thinking I was just returning to the original configuration, but what a mistake! Well, the old PC didn't have any issues - Windows XP just reused the nVidia driver, I just disabled autostart of the Catalyst, which was complaining about the missing ATI card.
Windows 7 in the new PC didn't have problems either - again, just reused the ATI driver, and I re-enabled the Catalyst autostart.
But completely different story with Ubuntu. It just didn't boot to the GUI! The monitors stayed black and were blinking with the power indicators, which meant there was no proper signal from the video card. Booting in recovery mode and attempt to reconfigure graphics didn't help. Neither the playing with xorg.conf. I removed everything related to nVidia - no luck. Removed ATI drivers as well - nothing... Then somewhere in the logs I noticed mentioning of VMWare and some warnings. I remembered that VMWare gets pretty deeply into the system, it even compiled some kernel modules during installation. So I decided to give it a try. Found how to uninstall it (followed instructions on this page: sudo vmware-installer -u vmware-player), and then after more playing with recovery mode that the system finally could boot successfully into X.
It wasn't the end though. There was no network connection! Apparently when VMWare was ununstalled, it took the network with it. More googling helped to find that "eth0" is not in /etc/network/interfaces anymore. Just added it to the first line ("auto lo eth0"), rebooted, and voilà - it worked!
Now I have just reinstalled ATI drivers, and re-configured the multi-display mode, and finally my computer is working again. No more experiments!