New Googlebot User-Agent Identification

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

[Critter]

Just noticed this tonight. A new identification for the Googlebot in my logs.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Verified that the IPs were Google's, so it seems legit. Funny that this new identification mimics Yahoo's crawler bot identification

[Net_Wizard]

Deep crawling have started.

[seofreak]

One of my sites has been getting deep crawled everyday since march 1. about 20 pages. <simpons mode: woohoo>

anyone else also experiencing this?

[plasma]

@critter:

Is the request really coming from google's IP-Range?

@seofreak:

we get daily fully crawled since a long time, even our PR4 sites.

[Critter]

Hi Plasma:

Yes, I've verified that the IPs are from Google's network allocation.

Furthermore, it's kind of obvious that the new bot is not a bogus crawler because it comes from different IPs within the same (Google) netblock, something that would be almost impossible for a scammer to pull off unless they had their own netblock (and they ain't givin' out Class Cs like they used to) :)

[nilloc]

Hi,

Just found a very strange entry in my Logs.
I am certain this is NOT GOOGLE
Someone else got this one also?

Host: 64.68.88.152
Url: /index.html
Http Code : 200
Date: Mar 05 13:05:37
Http Version: HTTP/1.1"
Size in Bytes: 16491
Referer: - Agent: Googlebot/Test

Regards,

[bull]

Yes, "Googlebot/Test" got two files.
IP was
64.68.89.144

Nilloc, the IP you mention IS Google, as well as the one I posted:

[edit] marcs was a little faster with the whois... :) [edit]

Any statment, Googleguy? Would be appreciated.

[edited by: bull at 6:35 am (utc) on Mar. 5, 2004]

[marcs]

>I am certain this is NOT GOOGLE
Maybe it is :

[rwhois.exodus.net]
network:Class-Name:network
network:Auth-Area:0.0.0.0/0
network:Network-Name:64.68.88.0
network:IP-Network:64.68.88.0/21
network:Organization;I:Google Inc.-BGPconfig-SC3DC3
network:Name;I:Google Inc.
network:Email;I:[e-mail]
network:Street;I:2400 E. Bayshore Pkwy
network:City;I:Mountain View , CA 94043
...

[nilloc]

Hi,

Thanks guys.

Changing my htaccess file Back again.
10-minutes ago I had blocked agent: Googlebot/Test

Letting it in again now

Regards,

[GoogleGuy]

"Any statement, Googleguy? Would be appreciated."

Absolutely, bull. Normally I post by the seat of my pants, but I wrote this up earlier today and hadn't posted it yet:

Hey everybody, I wanted to give you a heads-up about a potential change in our user-agent name. Currently we use the user-agent

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

but we're considering changing our user-agent to something like

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The primary reason for this is that some web servers assume that unless a user agent is IE, Netscape/Mozilla, or maybe Opera, that your browser won't support JavaScript, frames, etc. As Googlebot gets better over time, it gets closer to a regular user and browser in our ability to handle features like that. This user-agent change would let vanilla webservers assume Googlebot is more like a regular user, while still allowing site owners to recognize that Google is visiting and still providing clear contact info in case of questions or issues. We would still respect "Googlebot" in robots.txt, so the overwhelming majority of webmasters wouldn't have to change anything at all; most site owners probably wouldn't even notice the difference. This bot that a few people noticed was a test crawl with the new user agent. It looks like things went very smoothly, but we'll still be conservative when making this change. If people have any positive or negative comments or questions, just post here. If we don't hear any showstopping objections, we'll look at making this name change on the user-agent when we're convinced everything would be a smooth transition.

thanks,
GoogleGuy

[Dayo_UK]

Will this have an effect on how Googlebot treats Javascript and Frames?

Or is it purely a case on how the Web Server treats Googlebot?

Bit above me this type of talk :(

[steveb]

"allowing site owners to recognize that Google is visiting"

Not if they have Urchin stats.

[seofreak]

whoops

[Stefan]

But what is "Googlebot/Test"?

I have that in yesterday's logs from 64.68.89.144. It was getting a 406 from the server.

[superba]

I'm also getting 406 from 'Googlebot/Test' on pages that are fine (and already indexed for weeks.) Worrying. We haven't done a thing to them.

[Ledfish]

Only problem I see GG is that my stats vendor will have to update there software to recognize the new googlebot identification, but hey thats not a big deal and I can imagine that the negatives might outweight the good.

So the only thing I'd say is if it is all good, make a formal press release so that all the stats vendors will update our software so that we can know when googlebot is visiting and can see what you have looked at without having to resort to open our log files in something like excel and doing a "find" to pull up the googlebot entries.