It does sound like a document mode compatibility problem. That's something I haven't run into yet for the sites I work with, but I have read about it. Maybe this reference from Microsoft will provide a clue.
It's a bit buried on that MSDN page. Also if your (X)HTML is not valid then it may be taken as an excuse to force the page to be rendered in quirks mode. I personally stick to XHTML served as application/xhtml+xml because if the page breaks I know it on the spot and fix it within seconds. ;)