homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

GoogleBot User_Agent Changed
No longer using googlebot.com

 8:13 am on Jul 19, 2004 (gmt 0)

Since a couple of days I was wondering why my tracking system didn't register any hits from Googlebot. Then I discovered that the referrer ID in the logs did change from "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" to "Googlebot/2.1 (+http://www.google.com/bot.html)".

Just to let you know since I didn't notice any tread on this ...

cheers Panze



 1:57 pm on Jul 19, 2004 (gmt 0)

Thanks Panze, it looks like it was Tuesday or Wedensday last week.


 2:18 pm on Jul 19, 2004 (gmt 0)

[robotstxt.org...] needs update?

Patrick Taylor

 2:32 pm on Jul 19, 2004 (gmt 0)

It switched for me on 13th July (server in California).


 10:13 pm on Jul 19, 2004 (gmt 0)

Wow, the second change in G that resulted by the idea of one black hat WebmasterWorld member. Find out the other one as I don't want to mention the name. ;)


 8:23 am on Jul 20, 2004 (gmt 0)

There is no change needed in my robots files if I do not list this bot for any disallowments, is there?


 8:30 am on Jul 20, 2004 (gmt 0)

adfree, even if you do mention Googlebot in your /robots.txt you should have no problem.

When deciding which pages to crawl on a particular host, Googlebot will obey the first record in the robots.txt file with a User-Agent starting with "googlebot". If no such entry exists, it will obey the first entry with a User-Agent of "*".

It's the URI portion of the User_Agent header that's changed, so people using "user_agent cloaking" that relies on matching the entire string may be caught out by this.


 10:03 am on Jul 20, 2004 (gmt 0)

sweet that is so funny!


 10:22 pm on Jul 20, 2004 (gmt 0)

We've been talking about it for days.



 10:55 pm on Jul 20, 2004 (gmt 0)

Here's an even older thread with some comments by GoogleGuy:
that I found today when I noticed the change.

I was surprised to see it has been happening since March 3rd because I didn't see any change until July 9th.


 10:11 am on Jul 21, 2004 (gmt 0)

Thanks volatilegx, you guys certainly beat us to that one.

That's interesting yowza, I see the same as you, only later.


 3:59 am on Aug 15, 2004 (gmt 0)

What do I change

if ($ENV{'HTTP_USER_AGENT'} =~ /googlebot/) {

to? 7/28/04 is the last day that code generated any logs.


 4:22 am on Aug 15, 2004 (gmt 0)

*Tries changing it to

if ($ENV{'HTTP_USER_AGENT'} =~ /google/) {


and with in minutes Google shows up in the log!


 5:27 am on Aug 15, 2004 (gmt 0)

Jesse, I think that's because Googlebot has a capital "G", while www.google.com or www.googlebot.com is in lower case in Google's user_agent.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved