jdMorgan - 3:22 am on Sep 27, 2004 (gmt 0)

An HTML page would be seen as an invalid robots.txt file format, so the 'bot could decide that either
A) It should crawl the site
B) It should not crawl the site
The interpretation could vary between different 'bots, and even between different revisions of the same 'bot.

Don't waste time worrying about it, just remove the redirect and continue looking for any other problems, such as multiple subdomains resolving to the same pages, i.e. more that just domain.com and www.domain.com resolving to the same content.

After any change to robots.txt, and even just periodically, it's a good idea to validate [searchengineworld.com] it.

Jim