Msg#: 33049 posted 5:17 pm on Feb 17, 2006 (gmt 0)
The robots.txt page in Google Sitemaps is misleading because it makes it appear that Google's crawler was not able to find the robots.txt file.
I was confused (and concerned) because the page said I didn't have a robots.txt ("We didn't find a robots.txt file on your site.") even though I have one. Finally I discovered that clicking the "Check" button forces the utility to load the robots.txt file. The page is displaying information based on the tool's downloading of robots.txt, not Google's bots.