Google Michigan Location doing some Anti-Spam Checking?

[MLHmptn]

Recently in my logs I have been noticing Googlebot (ip location Alma, MI) scanning my site for files such as :

mx4000.free.download.html
download.wc3.1.20d.no.cd.html
6230i.theme.download.html
conter.strike.1.6v19.html

... and other seemingly crack, warez, etc. files. I do not host anything of this sort so I'm presuming Google is up to something or is banning hosts that serve up this kind of information, spyware, etc. Also noticing Yahoo spidering the same sort of files. 404 error pages being served up for responses but still...quite odd.

Anybody else noticing this in their logs or am I seemingly alone in this? About 8-9 months ago a server I admin was hacked via a PHPlivesupport exploit and had arbitrary base64 code planted on a site so possibly it was a previously hacked server?

[followgreg]

I've never seen such thing. Are you sure (and how do you know) it was Google?

But they really have a spam issue more than ever right now though.

[ashear]

Just for the heck of it I checked 3 months of my logs for that. I could not find any matches.

It is really easy to spoof your useragent when crawling people. I woudln't take it too seriuosly.

[netmeg]

Yea, I'd be pretty surprised if there's a Googlebot IP out of Alma Michigan. As mentioned above, it's easy to spoof a user agent, and try to convince people to serve you up content because you're Googlebot or Slurp.

[MLHmptn]

Well here are some of the log entries from my apache error_log file.

[Sat Aug 26 19:29:01 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/zuma.delux.with.keys.html
[Sat Aug 26 19:29:02 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/bajar.paint.shop.pro.8.html
[Sat Aug 26 19:29:05 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/resco.game.box.crack.html
[Sat Aug 26 19:29:06 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/www.crazyfrog.games.html
[Sat Aug 26 19:29:07 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/metatags/audio.editor.1.60.crack.html
[Sat Aug 26 19:29:07 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/includes/code.MP3.To.Ringtone.Pro.html
[Sat Aug 26 19:29:08 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/GRANITE.registration.key.html
[Sat Aug 26 19:29:15 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/metatags/descargar.Axel.2.0.gratis.html
[Sat Aug 26 19:29:16 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/free.serials.window.xp.pt.html
[Sat Aug 26 19:29:16 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/metatags/QuickTime.6.51.serial.key.html
[Sat Aug 26 19:29:17 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/metatags/Magic.Translator.v5.crack.html
[Sat Aug 26 19:29:18 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/activation.code,.crak.html
[Sat Aug 26 19:29:19 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/limewire.free.bg.warez.html
[Sat Aug 26 19:29:20 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/crack.3d.estudio.max.8.html
[Sat Aug 26 19:29:20 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/includes/hacker.upgrade.paltalk.8.2.html
[Sat Aug 26 19:29:21 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/metatags/6600.tomtom.mobile.cracked.html
[Sat Aug 26 19:29:23 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/key.generator.StyleXP.Male.html
[Sat Aug 26 19:29:24 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/metatags/download,msn.massenger.free.html
[Sat Aug 26 19:29:24 2006] [error] [client 66.249.65.80] File does not exist: /home/janusa/public_html/httpdocs/cachedir/age.of.castles.name.key.html

Notice that the 66.249.65.80 ip is from googlebot crawl, Alma, MI.

Very strange to say the least.

[edited by: MLHmptn at 5:11 pm (utc) on Sep. 11, 2006]

[SteveWh]

That is weird coming from Google. Maybe it has to do with the previous hack, or maybe something has got cross-indexed. Wouldn't be surprising.

Yahoo does, however, sometimes request files with nonsense names. They know the files shouldn't exist. They're checking to make sure your server properly returns a 404 code. But in that case, it should be basically nonsense names or random pages whose names seem to be from other sites. There shouldn't be a pattern such as the one you describe, where they all have to do with warez, cracking, and such.

[edited by: SteveWh at 5:32 pm (utc) on Sep. 11, 2006]

[ALbino]

Are you sure nobody is linking to you with those URLs? Or are you on virtual hosting with multiple people sharing the same IP?