GoogleBot User_Agent Changed

No longer using googlebot.com

[elmarpanzenberger]

Since a couple of days I was wondering why my tracking system didn't register any hits from Googlebot. Then I discovered that the referrer ID in the logs did change from "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" to "Googlebot/2.1 (+http://www.google.com/bot.html)".

Just to let you know since I didn't notice any tread on this ...

cheers Panze

[ciml]

Thanks Panze, it looks like it was Tuesday or Wedensday last week.

[Chndru]

[robotstxt.org...] needs update?

[Patrick Taylor]

It switched for me on 13th July (server in California).

[ThomasB]

Wow, the second change in G that resulted by the idea of one black hat WebmasterWorld member. Find out the other one as I don't want to mention the name. ;)

[adfree]

There is no change needed in my robots files if I do not list this bot for any disallowments, is there?

[ciml]

adfree, even if you do mention Googlebot in your /robots.txt you should have no problem.

[google.com...]

When deciding which pages to crawl on a particular host, Googlebot will obey the first record in the robots.txt file with a User-Agent starting with "googlebot". If no such entry exists, it will obey the first entry with a User-Agent of "*".

It's the URI portion of the User_Agent header that's changed, so people using "user_agent cloaking" that relies on matching the entire string may be caught out by this.

[DaveN]

sweet that is so funny!

[volatilegx]

We've been talking about it for days.

[webmasterworld.com...]

[yowza]

Here's an even older thread with some comments by GoogleGuy:
[webmasterworld.com...]
that I found today when I noticed the change.

I was surprised to see it has been happening since March 3rd because I didn't see any change until July 9th.

[ciml]

Thanks volatilegx, you guys certainly beat us to that one.

That's interesting yowza, I see the same as you, only later.

[Jesse_Smith]

What do I change

if ($ENV{'HTTP_USER_AGENT'} =~ /googlebot/) {

to? 7/28/04 is the last day that code generated any logs.

[Jesse_Smith]

*Tries changing it to

if ($ENV{'HTTP_USER_AGENT'} =~ /google/) {

*

and with in minutes Google shows up in the log!

[ciml]

Jesse, I think that's because Googlebot has a capital "G", while www.google.com or www.googlebot.com is in lower case in Google's user_agent.