64.233.173.67

Google spider

[Key_Master]

IP: 64.233.173.67
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AIRF; .NET CLR 1.1.4322; InfoPath.1)

I expect better from Google. Normally they are very good about robot behavior but this one is a little sneaky. Anybody have any idea what it's being used for? It's been pulling hundreds of pages a day from one of my sites. I finally banned it today till I get more info on it.

[volatilegx]

The "InfoPath.1" in the user agent is interesting. Anything to do with Microsoft's InfoPath product?

My guess is that this is not an automated spider.

[fiestagirl]

I believe that 64.233.172.* and 64.233.173.* are being used for the desktop search.
I usually have a referrer (not always GG), a legit UA, and the users usually request images and CSS.

The one I don't like is the one that is using this user agent: mozilla/4.0 (compatible; msie 6.0;)
216.239.58.*, 216.239.38.*, 64.233.166.*.
No referrer, no images, no CSS.

[Key_Master]

It's a spider. Even if it's 403'd it keeps crawling. It gives erroneous referrer info too. Most curious of all, it crawls the relative urls in forms as though it were an ordinary links but ignores the spider traps in the same directory. It crawls hundreds of pages a day throughout a 24 hour day (443 pages so far today).

I thought it was related to Google Desktop too. According to my records the same IP used Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Google-TR-1.67) a few months past.

Still, I can't figure out why this site would need to be spidered for Google Desktop. You have to understand, this has been happening daily for sometime now. I'm inclined to think something else is going on. I emailed Google. Don't expect an answer but they've surprised me before.

[Pfui]

G's been sneaking in (ignoring robots.txt) via numeric IP and a regular (non-Googlebot) UA quite a bit lately. And like you, I saw the same uptick in traffic this morning, too. Coming from --

64.233.173.73
64.233.173.80
64.233.173.86
64.233.173.89
64.233.173.106

-- and all those within the same *second*. The UA was always:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5

Lately, Jeeves has been sneaking in via numeric IP, but not as much as G(rrrrrr). I don't know why they're ignoring robots.txt and cloaking their IDs but I don't like it a bit.

[volatilegx]

Any chance this is related to the Google Web Accelerator?

[Key_Master]

Google Web Accelerator uses:
72.14.192.0 – 72.14.192.255

[Key_Master]

But, if it is a proxy or prefetch of some sort it's being abused. It's not a very smart program either. What kind of proxy pulls form urls?

I can see it now, banned from Adsense due to fraudulent clicks via visitors using a Google proxy.

[rebelde]

I certainly hope that, if G or anybody else legit is spidering without identifying themselves, they will realize that many websites ban lots of IP address of crawlers that download lots of pages.

[keyplyr]

My feeling is that it's most likely cooperate intranet, possibly running Google search hardware since InfoPath is showing in the string.

[volatilegx]

Yeah, with InfoPath in the string, and the fact that it is crawling form pages... sounds suspicious. Isn't InfoPath a MS XML application to develop forms?

[Kakaroto]

Volatilegx, it is indeed the Google Web Accelerator. It is working like a proxy and reporting their IP's instead of the customers'. I still don't understand why the computer is reporting real IP sometimes and other times the Google WA ones, we will have to read more about it. The fact is I had the Google WA turned on, and my own visits appeared in the web logs with those IP's, I turned off, and it went back to normal.

[Pfui]

I wonder if it/G is pre-fetching or pre-loading or post-tracking or who-knows-what-all because if real people are using G's IP(s) when they're on-site, I'm not hearing from them when they get blocked and redirected to an e-me page. Wait --

Come to think of it, they're not getting redirected. The IP 'users' hit a 302 and that's the last I see of them, at least for that session. But they don't act real.

Dang it. No ID, no 'proper' UA, no robots.txt, no referers, no redirects, and a hit rate too rapid for real people.

Gee. G cloaking. Go figure.

[volatilegx]

Thanks for the confirmation, Kakaroto.