Google-Site-Verification/1.0

[GaryK]

Google-Site-Verification/1.0
74.125.16.1
No PTR
-----
OrgName: Google Inc.
OrgID: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA

I've never seen this UA before. Google says it's used for verifying ownership of a site. Have any of you seen it before?

[incrediBILL]

I think it's a Google Webmaster Tools verification thing.

[Samizdata]

I have never seen that one, Webmaster Tools always used "Google-Sitemaps/1.0" before.

Still, without undocumented new bots every day we would have nothing to talk about.

...

[incrediBILL]

Not a site maps thing, it's when they check your validation ID installed on your site, a webmaster tools thing.

[dstiles]

Just got it on 74.125.75.4. Same UA, no rDNS, dodgy header fields. Blocked.

First sighting via google suggests April 11th.

Could be a verification meta tag checker - except the site it hit doesn't have that tag.

Project Honey Pot lists several IPs in the 74.125.nnn.nnn range as harvesters. :)

(edited by dstiles)

[GaryK]

Got hit from the same IP Address today with these UAs. Note the translate.google.com proxy:

Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+ru;+rv:1.9.0.8)+Gecko/2009032609+Firefox/3.0.8,gzip(gfe)+(via+translate.google.com)

Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+tr;+rv:1.9.0.5)+Gecko/2008120122+Firefox/3.0.5,gzip(gfe)+(via+translate.google.com)

Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+fr;+rv:1.9.0.9)+Gecko/2009040821+Firefox/3.0.9,gzip(gfe)+(via+translate.google.com)

[Samizdata]

Not a site maps thing, it's when they check your validation ID installed on your site

My records show that the "Google-Sitemaps/1.0" user-agent (from 74.125.16.xx) was always used to fetch my index page in the days when I used a META tag for the verification ID.

I now use an HTML file, and this is always fetched by the standard Googlebot.

Oddly enough, so is the actual sitemap.

...

[jmccormac]

Just seen this site verification UA on one of my sites. The IP was 64.233.172.2 which checks out as a Google IP.

Regards...jmcc

[dstiles]

Within half an hour this evening:

66.249.85.84
66.249.84.11
74.125.75.4
74.125.16.3
72.14.193.2
72.14.195.38
72.14.194.33

Probably valid since the site is registered with WMT. Reason for so many - new UA which got 403'd.

It would also really help if they kept specific IP ranges for specific functions.

[Pfui]

It checks a couple of different pairs of things. From this morning:

72.14.194.1
Google-Sitemaps/1.0

Date Page Status Referer
05/09 07:35:28 /google[siteverificationkey].html 200 -
05/09 07:35:28 /google[siteverificationkey].html 200 -
05/09 07:35:28 /noexist_[siteverificationkey].html 404 -
05/09 07:35:28 /noexist_[siteverificationkey].html 404 -

[dstiles]

I assume that means you have an external google key on your site instead of it being in the page header? I wouldn't be expecting that on our sites.

[Pfui]

Yep. I have a standalone verification file in that webspace. (Depending on engine and site/server specs, I use files for some keys, tags for others.)

[dstiles]

More of this today. Three sites hit, once per site. All sites have site verification registered with WMT. My beef is no rDNS. All three hits blocked.

UA: Google-Site-Verification/1.0
IP: 64.233.172.0-64.233.173.255
rDNS: none
Robots: haven't checked but I doubt it.

I suppose I should enable the range, though why I should when google stuff proxies and bots on the same IP ranges. I sometimes wonder if google actually knows anything about the internet at all. :(

[Pfui]

Enable the range, limited to the UA. Or limit the UA to the range. Etc.

Yahoo's verification-checker comes by a lot; Google's not so much. But repeated blocks means you'll eventually have to re-verify to be able to remove URLs and such.

(I know this for a fact because I've got an inadvertent block somewhere deep in .htaccess that's snagging "Yahoo! Slurp/Site Explorer". Alas, debugging is slow-going because Y's UA hails from a bunch of its different, legit hosts, e.g., inktomisearch.com and yahoo.net.)

[dstiles]

It really is well past time that all of the SEs sorted out their IP blocks with rDNS saying EXACTLY what each IP is used for. And preferably sort all bots of a common stripe into single ranges - or at least not intertwined with proxy scrapers.

I wonder if one could complain successfully to them that their verifier had no rDNS despite them stating publicly that their bots WOULD have rDNS set up AND that we should check it to verify the bots. Would they at least say sorry and fix the situation? No, thought not. :(

[GaryK]

What avenues are open to us for such complaints? I know I've seen use of Twitter hash tags work really well to get a company's attention via trending topics. Does GoogleGuy spend much time on WebmasterWorld anymore? What about their webmaster blog? It would have to be done carefully so as to avoid being seen as a call to action.

[dstiles]

What chance of them even noticing? Any of the SE's? All the "top" three are doing stupid things with bots. Some are doing stupid things with search results.

If there were more competition you could maybe expect them to try for an edge through appeasing site owners/managers but the only one with the nous to try is MS and they're worse at bot management than google, despite having the spur of a "new" engine.