Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Google stopped crawling many sites Jun 15 AM


seoisabusiness - 9:35 am on Jun 16, 2010 (gmt 0)


something strange we found, it's our first lead

but what we now found is

66.249.66.88 - - [16/Jun/2010:08:04:04 +0200] "GET /ticker/MBISetup.exe HTTP/1.1" 301 99 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.244 - - [16/Jun/2010:08:04:05 +0200] "GET /en/MBISetup.exe HTTP/1.1" 301 99 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.88 - - [16/Jun/2010:08:04:05 +0200] "GET /en/mbisetup.exe HTTP/1.1" 404 1236 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

for explanation, the /ticker/MBISetup.exe request is the original one, the others are redirects based on our "URL canonicalization" - logic.

on the other domains:

a .com domain
66.249.66.4 - - [16/Jun/2010:03:25:26 +0200] "GET /ticker/MBISetup.exe HTTP/1.1" 404 4258 "ref=-" "ua=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

on a .co.uk domain
66.249.66.129 - - [16/Jun/2010:03:59:08 +0200] "GET /ticker/MBISetup.exe HTTP/1.1" 404 3881 "ref=-" "ua=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

similar on other domains .fr, .at, .de, .....

the thing is, these request (which are ongoing in a 10 minutes plus on some domains, on a 1 minute schedule on some others) began shortly before google stopped crawling these sites, and intensified during the exact moment when googlebot stopped crawling completely. (the crawling stop is still ongoing).

we verified that these IPs are actual googlebot IPs.
there are no "malware alerts" in google webmaster tools visible on any of these domains.

we believe that these request
"GET /ticker/MBISetup.exe HTTP/1.1"
are googlebot checking for maleware (which is not on our site) but something went wrong an googlebot sees our site as maleware - without flagging it?

this is very strange behavior and we see it over multiple domains (different servers, different datacenters, different codebase, different companies).

would be great if somebody checks their logfiles for similar requests - or has an explanation what is going on.


Thread source:: http://www.webmasterworld.com/google/4152972.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com