New Google BOT?

[MonkeyAllen]

Okay guys, i really think google unleased a new bot into the web sometime this week. For 9 months now MonkeyReview has been dealing with issues with the google bot not being able to crawl our reviews, always mixing up the ? with 3%F, the other day I noticed low and behold google crawling at least one of the reviews, perhaps my mind went into instant overdrive from the excitment but the fact that it said Google Bot 2.1 struck me as funny, anyone else notice this?

Chris Allen
<snip>

[edited by: ciml at 5:23 pm (utc) on July 19, 2002]
[edit reason] No .sigs please. [/edit]

[jaytierney]

I recall a few other people mentioning that some of their generated pages were being crawled either for the first time or more frequently. Perhaps the Google Bot has matured into an expert link seeker and has overcome some of its past limitations? :)

[ciml]

This is the oldest Googlebot hit I can find right now, it's from May 2001 and the UA is Googlebot/2.1. I'm sure someone will have better records than me.

216.239.46.183
-
-
[17/May/2001:16:13:27 +0100]
"GET /castle/ HTTP/1.0"
200
3618
"-"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

[Key_Master]

I think it was a human on your site (possibly a Google employee). I've noticed that Google will sometimes use different user agents while running tests. Sometimes the user agent will simply be Googlebot/2.1 although I believe they can make it whatever user agent they wish it to be.

[MonkeyAllen]

This is what i could drug out of one of my stats prgrams, notice it doesn't reference the googlebot 2.1 but only 2, these are a day old set of stats so as the day progresses i should see soemthing in my other programs.

15: 16016: 1427: Googlebot
: 16016: 1427: Googlebot/2

[HyperGeek]

I'll tell you this much, I changed the TITLE of my site and the title in the listing changed the next day.

It then changed back to the original the day after that, but it shows that the googlebot is getting more aggressive.

I've been visited 10 times this week for some reason.

[taxpod]

Almost all of my Googlebot hits in the past few months, and I get a lot of them, are from Googlebot/2.1. there is no way that this is a human - the ones on my site - as the frequency is too great. Analog reports the traffic as Googlebot/2 by the way.

[Key_Master]

Taxpod,

Do you mean

Googlebot/2.1

or

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

There's a big difference between the two agents.

I think Google omits the "+" character from the user agent while running tests because some server side scripts are not configured to handle this character properly (which is why I think they continue to use it).

<personal opinion>Stat programs should report the full user agent.</personal opinion>