Welcome to WebmasterWorld Guest from 54.159.190.106

Forum Moderators: open

Message Too Old, No Replies

GoogleBot User_Agent Changed

No longer using googlebot.com

   
8:13 am on Jul 19, 2004 (gmt 0)

10+ Year Member



Since a couple of days I was wondering why my tracking system didn't register any hits from Googlebot. Then I discovered that the referrer ID in the logs did change from "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" to "Googlebot/2.1 (+http://www.google.com/bot.html)".

Just to let you know since I didn't notice any tread on this ...

cheers Panze

1:57 pm on Jul 19, 2004 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Thanks Panze, it looks like it was Tuesday or Wedensday last week.
2:18 pm on Jul 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



[robotstxt.org...] needs update?
2:32 pm on Jul 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It switched for me on 13th July (server in California).
10:13 pm on Jul 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Wow, the second change in G that resulted by the idea of one black hat WebmasterWorld member. Find out the other one as I don't want to mention the name. ;)
8:23 am on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There is no change needed in my robots files if I do not list this bot for any disallowments, is there?
8:30 am on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



adfree, even if you do mention Googlebot in your /robots.txt you should have no problem.

[google.com...]

When deciding which pages to crawl on a particular host, Googlebot will obey the first record in the robots.txt file with a User-Agent starting with "googlebot". If no such entry exists, it will obey the first entry with a User-Agent of "*".

It's the URI portion of the User_Agent header that's changed, so people using "user_agent cloaking" that relies on matching the entire string may be caught out by this.

10:03 am on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



sweet that is so funny!
10:22 pm on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We've been talking about it for days.

[webmasterworld.com...]

10:55 pm on Jul 20, 2004 (gmt 0)

10+ Year Member



Here's an even older thread with some comments by GoogleGuy:
[webmasterworld.com...]
that I found today when I noticed the change.

I was surprised to see it has been happening since March 3rd because I didn't see any change until July 9th.

10:11 am on Jul 21, 2004 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Thanks volatilegx, you guys certainly beat us to that one.

That's interesting yowza, I see the same as you, only later.

3:59 am on Aug 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



What do I change

if ($ENV{'HTTP_USER_AGENT'} =~ /googlebot/) {

to? 7/28/04 is the last day that code generated any logs.

4:22 am on Aug 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



*Tries changing it to

if ($ENV{'HTTP_USER_AGENT'} =~ /google/) {

*

and with in minutes Google shows up in the log!

5:27 am on Aug 15, 2004 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Jesse, I think that's because Googlebot has a capital "G", while www.google.com or www.googlebot.com is in lower case in Google's user_agent.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month