Googlebot user-agent string changed

sometime in the past day

[jdMorgan]

Just noted the switchover:

64.68.82.14 - - [13/Jul/2004:05:13:10 -0400] "GET / HTTP/1.0" 200 37577 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 
  
64.68.82.182 - - [14/Jul/2004:01:27:14 -0400] "GET / HTTP/1.0" 403 670 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 

This makes it rather hard to control the Google-spoofers without shooting your foot off... :(

Jim

[Josk]

Well... I've seen "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" since 3rd March 2004 and "Googlebot/2.1 (+http://www.google.com/bot.html)" since 15 June 2004, so this is pretty old news.

[amznVibe]

Somewhere on the web, hundreds, maybe thousands of sites just became uncloaked :)
They will be caught and pulled from the SERPS.
This is a *good* thing!

[TheDave]

Thats exactly what I was thinking amznVibe. Good work google. I don't know why they don't send a spider out "cloaked" as IE ;) I guess for all we know, they do..

[fiestagirl]

Yes we all know that they do and this is the UA:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/2003070

[bcolflesh]

Yes we all know that they do and this is the UA

You have that UA coming from the established Googlebot ranges?

[Clark]

Aren't there some legitimate sites using cloaking in a legitimate way? Like IPB uses cloaking to deliver google a navigation that allows them to view pages correctly? Are they misguided?

[jdMorgan]

They might get pulled from the SERPs if they are serving substantially-different content with intent to deceive, but the reason I noticed this in the first place is that one site I administrate was getting so badly abused that it is now "armored"; It won't allow access from any unrecognized user-agent. When the new Google user-agent visited, it got a 403-Forbidden.

Since it's a non-profit info site only, this is no big deal. But of course I fixed the problem by allowing the new user-agent. Googlebot came back the next day, and the site didn't suffer in the SERPs at all.

Additionally, the site *is* cloaked in that it returns a shortened, "prettier" title to search engines that do stemming for plurals like Google does. Still -- no ill effects from this situation.

Maybe this *is* old news, but I didn't see any other posts on it.

Jim