There's nothing wrong with that robots.txt file, since a blank "Disallow" disallows nothing: # Disallow all spiders from *no* pages (i.e. "Allow all") User-agent: * Disallow:
# Disallow all spiders from *all* pages (i.e. "Allow none") User-agent: * Disallow: /
You might want to check your robots.txt file with a different validator, and also check it with a server headers checker and a user-agent spoofing tool (spoof as Google, and see what you get). robot.txt must be a plain-text (ASCII) file with a MIME-type of "text/plain", and must be located in the Web root directory (where your "home" page is located), so make sure that's where it is.
Note that you could serve a blank robots.txt or delete it entirely to achieve the same "Allow all" result as the one you're using now.
Nothing changed (everything with the sitemaps was OK and then last night I noticed on the Google sitemap both sites had the red ERROR and defined it as " URL restricted by robots.txt ". My robots.txt file allows all robots and all pages as written above. This same exact robots.txt file worked fine for a long time.
I saw in another post someone recommended removing the .htaccess file just in case something quirky was happening so I did that, re-submitted the sitemap and an hour later it crawled it and came back with the same error.
Just in case I reloaded the robots.txt file (ASCII text mode as always) and that did not correct the problem...
Msg#: 34058 posted 12:01 am on Apr 27, 2006 (gmt 0)
AND I will add this: Google's own robot.txt tool says OK when I test the site url + sitemap.xml to check this. SO, the problem is GOOGLE'S Erroneous ERROR message on the actual sitemap page. And it started with 2 sites which had not been changed.
Msg#: 34058 posted 12:04 am on Apr 27, 2006 (gmt 0)
UPDATE: I just found on the Google Groups message page the following:
"Several of you have noticed an issue with robots.txt reporting. Thanks for letting us know about this. We have fixed this issue and you should see updated status in your Sitemap account shortly. Our latest blog post contains more information. by Google Employee - 5:41pm"