New Googlebot User-Agent Identification

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

[Critter]

Just noticed this tonight. A new identification for the Googlebot in my logs.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Verified that the IPs were Google's, so it seems legit. Funny that this new identification mimics Yahoo's crawler bot identification

[kaled]

Presumably this is a significantly more powerful robot rather than just a new name.

GG, you mentioned frames and javascript - well I don't envy your engineers the task of dealing with javascript (so I guess that may be down the line) but since I use frames extensively I would certainly be interested to know what changes are coming.

I imagine that CSS files will also be scanned in the future. I guess we'd all like the head's up on that.

Also here's a suggestion I posted a couple of weeks ago.

From comments by GoogleGuy, I think it is safe to assume that it is possible to create plain html links that will not be followed by Googlebot. All you need to do is add something that looks like a session id to the url.

However, this is untidy, therefore I propose a very simple exclusion protocol just add

?...&robots=nofollow

to the url.

When a robot sees this parameter in an url it should not follow it.

The standard should allow other fields and fields in any order so that the following would be legal

?...&robots=newparam,nofollow,anotherparam

This would make it easy for webmasters to avoid setting spider traps. It would allow creators of shopping cart software to ensure that their products don't set spider traps. Since it is probably the existence of such problems that has caused some hosts to ban Googlebot (amongst others) it would help to solve this problem over time.

A standard such as this should have been agreed years ago. However, if we wait for a standards organisation to ratify this it'll take years. On the other hand, if Google were to unilaterally adopt such a standard, other robots would adopt it too.

Kaled.

[GoogleGuy]

Interesting suggestion, kaled--I'll pass it on. You could also imagine allowing a div that lets you block out links or sections of a page not to index/follow. Thanks for the feedback, everybody.

Dayo_UK, it's more the latter. It's not like the new user-agent bot will be some brand-new "superbot" that can understand everything that webservers will offer. But it does lay the groundwork, so that if in the future we want to add a new superbot-like feature, things will be smoother for everyone (both webmasters and us).

[Stefan]

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html

Fine and dandy, but what is this?

2004-03-05 03:37:30 64.68.89.144 GET /b*rrows_icc_030510.htm 406 4085 134 www.site.org Googlebot/Test -

Did I miss something in the thread or did it address it? What's with the 406?

[ScottM]

GG, a lot of us are running session 'killers' ("cloaking" for a good reason) for Googlebot on our forums.

Will this affect us?

If so, should we change our code to the new name?

[edited by: ScottM at 2:50 am (utc) on Mar. 6, 2004]

[jbgilbert]

If that 406 is the response to the user agent it means:
Client Error - Not Acceptable.

Very interesting log entry you have there...

[Stefan]

It's the first 406 I've ever seen in the logs. I don't know what the test is, but it's not working with my site.

[jdMorgan]

We've had some recent discussion of the 406 error over in the Apache forum. It seems there may have been some change recently that affected Googlebot's ability to participate in content negotiation. If you have content negotiation enabled but are not actually using it, you can turn it off to fix this problem.

On Apache, the fix can be as simple as putting

 Options -MultiViews 

in your Web root .htaccess file.

Jim

[Stefan]

JD, it's as simple an html page as you can get, the exact same as a hundred others on the site, many of which were crawled by the normal bot during the same 24 hr period and all got 200's.

I'll sticky you the log files if you want.

Interesting that GG didn't respond to the several posts on Googlebot/Test

[sblake]

"This bot that a few people noticed was a test crawl with the new user agent."

It seemed pretty clear to me.

[Stefan]

The test crawl he refered to wasn't Googlebot/Test and it wasn't getting 406's from servers. Read through the thread.

Added: this is what he was talking about

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

[edited by: Stefan at 3:39 am (utc) on Mar. 6, 2004]

[jdMorgan]

stefan,

Multiviews is a server-level setting, independent of (well, above) individual pages. You may have to ask your host if it's enabled. Basically, your server and Googlebot could not agree on a MIME type that was acceptable to both. You might also want to check your server headers [webmasterworld.com] and make sure they're correct. You should get a MIME-type of text/html for a plain-vanilla html page.

The problem with Googlebot and MultiViews started about 01/Feb/2004 according to this thread [webmasterworld.com].
More recent discussion [webmasterworld.com].

Jim

[Stefan]

Thanks JD

Our site gets crawled by the normal bot very well every day. The 406 only showed up on two attempts by that particular bot with that particular IP#. Whatever it is, it's not our site or server that caused the 406..

[lunas]

So, I for one am unclear, what is this GoogleBot/Test? It hit my site and received 200 meaning all is okay, but curious as to what it is.

[GoogleGuy]

I hadn't heard of GoogleBot/Test, but I'll ask about the 406 issue. I wouldn't be surprised to see different Googlebots with slightly different code in some ways--we're always trying new things. ScottM, I'd say it's best to prepare to recognize Googlebot by looking for either the old or the new user agent.

[volatilegx]

FYI, I have already seen the "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html) User agent" and was seeing it as early as late February. I too am curious about Googlebot/Test