I recently updated the initial version of a basic statistics script to start recording various headers. I firmly believe in serving the same content to humans and spiders however since my site is XHTML (served correctly as application/xhtml+xml) there are a lot of issues I have to deal with; mainly IE's lack of support (IE9 now supports XHTML though) and the questionable support among legitimate search spiders.
The main obstacle to serving pages universally as XHTML (once IE8 and older market shares are beyond even business's cryptic desire for support) is the lack of HTTP ACCEPT header being set or too vague. Currently the only major search spider (at least here in the United States over the past few days) to correctly serve the HTTP ACCEPT header is Yahoo. Most spiders that do have the HTTP ACCEPT header set only declare support for "*/*" which is completely unusable as there are many instances of Internet Explorer serving this as the entire
HTTP ACCEPT string. For XHTML sites to correctly serve XHTML as application/xhtml+xml it is absolutely necessary to detect the string "application/xhtml+xml" instead of the HTTP ACCEPT header.
I hope that someone over at Microsoft could forward this information to the folks running MSNbot so if it does understand XHTML as application/xhtml+xml then they could set the HTTP ACCEPT header to include the string. Here is an example of a browser's HTTP ACCEPT header...
Remember, it is absolutely necessary to see the string application/xhtml+xml
in the header otherwise even if */* is present the server has to presume the user agent only understands text/html due to Internet Explorer using only