New Googlebot User-Agent Identification

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

[Critter]

Just noticed this tonight. A new identification for the Googlebot in my logs.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Verified that the IPs were Google's, so it seems legit. Funny that this new identification mimics Yahoo's crawler bot identification

[g1smd]

I have seen 406 errors from another bot in the last few days.

2004-03-06 11:06:59 wfp2.almaden.ibm.com - W3xxnxnnn Wxxnnn ip.232.131.ip 80
GET /pdfs/Some_File.pdf - 406 0 4066 195 HTTP/1.0
[almaden.ibm.com...] -

.

Heh, even found someone browsing the site from a mobile phone.

2004-03-06 11:34:21 216.239.39.5 - W3xxnxnnn Wxxnnn ip.232.131.ip 80
GET /index.htm - 200 0 1069 565 HTTP/1.0
Nokia3510i/1.0+(04.01)+Profile/MIDP-1.0+Configuration/CLDC-1.0+(Google+WAP+Proxy/1.0) -

[edit_g]

Now, what does the + do? :)

[Stefan]

Interesting, g1smd. I also had them from the msnbot but it was on .doc's and .txt's, (but maybe I'm noticing them now because I'm looking for them).

2004-03-05 06:11:01 65.54.188.8 GET /c***-r****-a**essment.doc 406 4066 216 www.site.org msnbot/0.11+(+http://search.msn.com/msnbot.htm) -

The G ones were .htm.
The testbot and the 406's haven't shown up again, so no problem.

Thanks for the input, GG.

[mr_strong]

Googleguy - Are these changes to Googlebot the reason why many new sites aren't being deepcrawled at the moment?

Thanks :)

[seofreak]

one of my sites have been getting deep crawled daily.

[lstrand]

Any more ideas if the user agent "Googlebot/Test (+http://www.googlebot.com/bot.html)" is a real Google bot or a fake? There didn't appear to be a definitive answer in this thread.

This user agent really upped the page views starting last night and it is going for our regular HTML pages. In the past it had a few crawls and now it is far more aggressive for its visit cycle. It is also visiting new pages it has not crawled before.

I'm concerned about non-Google scraping of pages. If a fake, I'd like to ban it.

Here are the IPs and view counts (past month) identifying itself as the "Google/Test" bot.

IP Page Views
64.68.91.131 106
64.68.91.53 54
64.68.91.164 53
64.68.81.18 50
64.68.91.54 34
64.68.89.141 14
64.68.91.33 12
64.68.91.59 12
64.68.83.188 7
64.68.89.167 6
64.68.91.64 6
64.68.83.79 5
64.68.91.188 5
64.68.81.154 3
64.68.89.154 3
64.68.83.132 2
64.68.83.182 2
64.68.88.18 2
64.68.88.191 2
64.68.89.173 2
64.68.89.180 2
64.68.91.39 2
64.68.83.144 1
64.68.89.179 1

DNS lookups results are unclear. I see Exodus and Google for most IPs. Is it really Google?

Cable & Wireless SC3-1 (NET-64-68-64-0-1)
64.68.64.0 - 64.68.95.255
Google Inc. EC12-1-GOOGLE (NET-64-68-80-0-1)
64.68.80.0 - 64.68.87.255

Who has answers? The help would be most appreciated.

[GoogleGuy]

My hunch is that it's a real Google bot.

[Spine]

What's the best way to ban all googlebots old and new from a site using robots.txt?

[Dayo_UK]

If you really want to:-

User-agent: Googlebot
Disallow: /

BTW, Googlebot test is going mad at the moment on my site too - perhaps the word test will be dropped soon :)