homepage Welcome to WebmasterWorld Guest from 54.196.63.93
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google Site Map robots.txt error
robots.txt error in google sitemaps
scubajared

5+ Year Member



 
Msg#: 34058 posted 4:54 pm on Apr 26, 2006 (gmt 0)

When I logged into my Google Sitemap account today I found this message on the main page.

We can't currently access your home page because of a robots.txt restriction.

The robots text I use is

User-agent: *
Disallow:

Would this in anyway stop the Google spiders from index my homepage?

There seems to be other people having the same problem according to the Google sitemap forum.

Has anyone else seen this in their Google sitemaps account?

Thanks

 

suzie250

5+ Year Member



 
Msg#: 34058 posted 6:00 pm on Apr 26, 2006 (gmt 0)

You are disallowing all robots that way. Read up on robots.txt in the robots forum or at robotstxt.org

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34058 posted 6:12 pm on Apr 26, 2006 (gmt 0)

There's nothing wrong with that robots.txt file, since a blank "Disallow" disallows nothing:

# Disallow all spiders from *no* pages (i.e. "Allow all")
User-agent: *
Disallow:

# Disallow all spiders from *all* pages (i.e. "Allow none")
User-agent: *
Disallow: /

You might want to check your robots.txt file with a different validator, and also check it with a server headers checker and a user-agent spoofing tool (spoof as Google, and see what you get). robot.txt must be a plain-text (ASCII) file with a MIME-type of "text/plain", and must be located in the Web root directory (where your "home" page is located), so make sure that's where it is.

Note that you could serve a blank robots.txt or delete it entirely to achieve the same "Allow all" result as the one you're using now.

Ref: [robotstxt.org...]

Jim

abates

10+ Year Member



 
Msg#: 34058 posted 10:13 pm on Apr 26, 2006 (gmt 0)

Did you try putting your root URL into the form at the bottom of the page to see if Google thought it was blocked?

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34058 posted 10:15 pm on Apr 26, 2006 (gmt 0)

>>Did you try putting your root URL into the form at the bottom of the page to see if Google thought it was blocked?

By default, Google tells you if you're blocking robots.txt

Duckman2000

10+ Year Member



 
Msg#: 34058 posted 11:48 pm on Apr 26, 2006 (gmt 0)

I am having the same problem with 2 of my sites.

Nothing changed (everything with the sitemaps was OK and then last night I noticed on the Google sitemap both sites had the red ERROR and defined it as " URL restricted by robots.txt ". My robots.txt file allows all robots and all pages as written above. This same exact robots.txt file worked fine for a long time.

I saw in another post someone recommended removing the .htaccess file just in case something quirky was happening so I did that, re-submitted the sitemap and an hour later it crawled it and came back with the same error.

Just in case I reloaded the robots.txt file (ASCII text mode as always) and that did not correct the problem...

This is odd...

Duckman2000

10+ Year Member



 
Msg#: 34058 posted 12:01 am on Apr 27, 2006 (gmt 0)

AND I will add this: Google's own robot.txt tool says OK when I test the site url + sitemap.xml to check this. SO, the problem is GOOGLE'S Erroneous ERROR message on the actual sitemap page. And it started with 2 sites which had not been changed.

Duckman2000

10+ Year Member



 
Msg#: 34058 posted 12:04 am on Apr 27, 2006 (gmt 0)

UPDATE: I just found on the Google Groups message page the following:

"Several of you have noticed an issue with robots.txt reporting. Thanks for letting us know about this. We have fixed this issue and you should see updated status in your Sitemap account shortly. Our latest blog post contains more information.
by Google Employee - 5:41pm"

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34058 posted 1:47 am on Apr 27, 2006 (gmt 0)

Duckman2000 -

I've been using Google Sitemap for about five months now. It's still got some bugs, but it's definately improving.

For example, I got a Network Unreachable error, resubmitted the sitemap and within minutes the error disappeared.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved