| 6:00 pm on Apr 26, 2006 (gmt 0)|
You are disallowing all robots that way. Read up on robots.txt in the robots forum or at robotstxt.org
| 6:12 pm on Apr 26, 2006 (gmt 0)|
There's nothing wrong with that robots.txt file, since a blank "Disallow" disallows nothing:
# Disallow all spiders from *no* pages (i.e. "Allow all")
# Disallow all spiders from *all* pages (i.e. "Allow none")
You might want to check your robots.txt file with a different validator, and also check it with a server headers checker and a user-agent spoofing tool (spoof as Google, and see what you get). robot.txt must be a plain-text (ASCII) file with a MIME-type of "text/plain", and must be located in the Web root directory (where your "home" page is located), so make sure that's where it is.
Note that you could serve a blank robots.txt or delete it entirely to achieve the same "Allow all" result as the one you're using now.
| 10:13 pm on Apr 26, 2006 (gmt 0)|
Did you try putting your root URL into the form at the bottom of the page to see if Google thought it was blocked?
| 10:15 pm on Apr 26, 2006 (gmt 0)|
>>Did you try putting your root URL into the form at the bottom of the page to see if Google thought it was blocked?
By default, Google tells you if you're blocking robots.txt
| 11:48 pm on Apr 26, 2006 (gmt 0)|
I am having the same problem with 2 of my sites.
Nothing changed (everything with the sitemaps was OK and then last night I noticed on the Google sitemap both sites had the red ERROR and defined it as " URL restricted by robots.txt ". My robots.txt file allows all robots and all pages as written above. This same exact robots.txt file worked fine for a long time.
I saw in another post someone recommended removing the .htaccess file just in case something quirky was happening so I did that, re-submitted the sitemap and an hour later it crawled it and came back with the same error.
Just in case I reloaded the robots.txt file (ASCII text mode as always) and that did not correct the problem...
This is odd...
| 12:01 am on Apr 27, 2006 (gmt 0)|
AND I will add this: Google's own robot.txt tool says OK when I test the site url + sitemap.xml to check this. SO, the problem is GOOGLE'S Erroneous ERROR message on the actual sitemap page. And it started with 2 sites which had not been changed.
| 12:04 am on Apr 27, 2006 (gmt 0)|
UPDATE: I just found on the Google Groups message page the following:
"Several of you have noticed an issue with robots.txt reporting. Thanks for letting us know about this. We have fixed this issue and you should see updated status in your Sitemap account shortly. Our latest blog post contains more information.
by Google Employee - 5:41pm"
| 1:47 am on Apr 27, 2006 (gmt 0)|
I've been using Google Sitemap for about five months now. It's still got some bugs, but it's definately improving.
For example, I got a Network Unreachable error, resubmitted the sitemap and within minutes the error disappeared.