That's Strange - Googlebot Trips my Bot Trap

[Edge]

I was just looking through my error_log file and noticed a familiar denied by 403 ip address. When I did a "Whois" it turned out to be the googlebot (64.233.173.67). A little research shoes that it hit one of my long standing bot traps (three+ years old). I looked in my Robots.txt and the trap was listed. - Strange, has anybody else seen the googlebot ignore the robots.txt?

First for me...

[montefin]

Edge,

Since about April 11, 2006, googlebot has been experiencing "issues" with robots.txt files.

I know because Google's "URLs restricted by robots.txt" tool showed 3 of my URLs blocked even though my robots.txt was wide open at the time.

Subsequently, I read in another post here on webmasterworld that the tool can only report 3 URLs -- ergo, there could be quite a few more.

From April 12th on, Google Analytics shows my overall Google Referrals, down by 46% -- by 96% on important AdSense keywords. Could be this, could be something else. Lot going on with Google right now.

You can read the post by Vanessa Fox of Google Engineering at the Inside Google Sitemaps blog. Hopefully the link for that -- [sitemaps.blogspot.com...] -- will be okay by webmasterworld. If not sticky me and I'll give it to you.

[prieshach]

<< Subsequently, I read in another post here on webmasterworld that the tool can only report 3 URLs >>

I currently have 7 urls showing (correctly) under 'URLs restricted by robots.txt'. It varies depending on recent bot activity.

[montefin]

prieshach,

Thanks for the correction. It's a relief.

[abates]

A couple of months back, Google stopped being able to "see" my robots.txt. In sitemaps, it reported that I didn't have one, and restricted URLs started showing up in search results. This came right after a time...

[kilonox]

I don't see a reason to use a robots.txt really...

// my top secret bot
if (robots.txt exists and is world readable){
 **parse the file, go ahead and read allow sections now
 **later grab disallows from the special bot IP address
 **yada yada
 **bunch of code
 **silly webmasters
} else { 
  **just rip the site
}

[Edge]

Ok , this drama is getting interesting. I have now banned a google bot three times. Once yesterday, and this morning, all different ip's. I have looked my robots.txt over and cannot find a thing wrong with it. I'm amazed that the googlebot acts like every other bot when it encounters a 403..

Should I inform GG?