Google Site Map robots.txt error

robots.txt error in google sitemaps

[scubajared]

When I logged into my Google Sitemap account today I found this message on the main page.

We can't currently access your home page because of a robots.txt restriction.

The robots text I use is

User-agent: *
Disallow:

Would this in anyway stop the Google spiders from index my homepage?

There seems to be other people having the same problem according to the Google sitemap forum.

Has anyone else seen this in their Google sitemaps account?

Thanks

[suzie250]

You are disallowing all robots that way. Read up on robots.txt in the robots forum or at robotstxt.org

[jdMorgan]

There's nothing wrong with that robots.txt file, since a blank "Disallow" disallows nothing:

# Disallow all spiders from *no* pages (i.e. "Allow all")
User-agent: *
Disallow:
 
# Disallow all spiders from *all* pages (i.e. "Allow none")
User-agent: *
Disallow: /

You might want to check your robots.txt file with a different validator, and also check it with a server headers checker and a user-agent spoofing tool (spoof as Google, and see what you get). robot.txt must be a plain-text (ASCII) file with a MIME-type of "text/plain", and must be located in the Web root directory (where your "home" page is located), so make sure that's where it is.

Note that you could serve a blank robots.txt or delete it entirely to achieve the same "Allow all" result as the one you're using now.

Ref: [robotstxt.org...]

Jim

[abates]

Did you try putting your root URL into the form at the bottom of the page to see if Google thought it was blocked?

[BillyS]

>>Did you try putting your root URL into the form at the bottom of the page to see if Google thought it was blocked?

By default, Google tells you if you're blocking robots.txt

[Duckman2000]

I am having the same problem with 2 of my sites.

Nothing changed (everything with the sitemaps was OK and then last night I noticed on the Google sitemap both sites had the red ERROR and defined it as " URL restricted by robots.txt ". My robots.txt file allows all robots and all pages as written above. This same exact robots.txt file worked fine for a long time.

I saw in another post someone recommended removing the .htaccess file just in case something quirky was happening so I did that, re-submitted the sitemap and an hour later it crawled it and came back with the same error.

Just in case I reloaded the robots.txt file (ASCII text mode as always) and that did not correct the problem...

This is odd...

[Duckman2000]

AND I will add this: Google's own robot.txt tool says OK when I test the site url + sitemap.xml to check this. SO, the problem is GOOGLE'S Erroneous ERROR message on the actual sitemap page. And it started with 2 sites which had not been changed.

[Duckman2000]

UPDATE: I just found on the Google Groups message page the following:

"Several of you have noticed an issue with robots.txt reporting. Thanks for letting us know about this. We have fixed this issue and you should see updated status in your Sitemap account shortly. Our latest blog post contains more information.
by Google Employee - 5:41pm"

[BillyS]

Duckman2000 -

I've been using Google Sitemap for about five months now. It's still got some bugs, but it's definately improving.

For example, I got a Network Unreachable error, resubmitted the sitemap and within minutes the error disappeared.