homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Strange Incident with robots.txt
error code 401

10+ Year Member

Msg#: 57 posted 6:57 am on Feb 16, 2003 (gmt 0)

I have had a very strange incident with googlebot and scooter.

They both came to my site one by one (next to each other) and looked for robots.txt 3 days ago.

They could not access the file and got error code 401 in return......:(( i don't know how, but I fixed my fault. I hope the permissions are set now...At least I can see the file now....I verified it too....

The question is, how does ERROR 401affect GoogleBot and Scooter? What is their Reaction?

Please help.....



WebmasterWorld Senior Member 10+ Year Member

Msg#: 57 posted 8:22 am on Feb 17, 2003 (gmt 0)

Wow, I'm not sure about this one.

There are two possibilities.

1. The bot will ignore the bad request and treat this like a 404 error. Since it cannot find or read the robots.txt, nothing is disallowed and the bot will proceed to request your regular pages.

2. The bot will recognize that the 401 error is caused by a page that requires authorization, and will treat this site as if it is off limits. Under this scenario, the bot will just leave.

I'm inclined to believe that the bot will react according to the first scenario.

The question is, did the bot grab any other pages besides the robots.txt? Check your logs.


10+ Year Member

Msg#: 57 posted 4:57 pm on Feb 17, 2003 (gmt 0)

I checked my logs,

The bots had grabbed a lot of pages BEFORE it hit the 401 error (the earlier days). One day, it got 401 for robots.txt, and left without grabbing ANYpage.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved