It does sound like a document mode compatibility problem. That's something I haven't run into yet for the sites I work with, but I have read about it. Maybe this reference from Microsoft will provide a clue.
You can force IE (regardless of which version) to always use it's best/latest version of the Trident rendering engine by using the following meta element...
It's a bit buried on that MSDN page. Also if your (X)HTML is not valid then it may be taken as an excuse to force the page to be rendered in quirks mode. I personally stick to XHTML served as application/xhtml+xml because if the page breaks I know it on the spot and fix it within seconds. ;)