downforeveryoneorjustme Revisited

(incl. AppEngine-Google)

[Pfui]

----------
CURRENTLY
----------

Note simultaneous timing of three latter hits: IP switching on the fly. Earliest hit also just to favicon.

74.125.75.3 [Google Mountain View]
AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
05/04 05:44:18 /favicon.ico
05/04 07:22:18 /
05/04 07:22:19 /

robots.txt? NO

64.233.172.17 [Google Shepherdsville]
AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
05/04 07:22:23 /

robots.txt? NO

----------
PREVIOUSLY
----------

Nov 15, 2009
AppEngine-Google | now used by Power Twitter [webmasterworld.com...]

May 10, 2009
AppEngine-Google: New Google UA to watch for [webmasterworld.com...]

Jul 31, 2008
AppEngine-Google | New bot from google [webmasterworld.com...]

----------
THOUGHTS
----------

The downforeveryoneorjustme site sits on and runs from Google IPs but is registered to an SEO consultant who also owns the hosting company spamvertised in the result(s). Not sure how a for-profit company gets to piggyback an app onto Google for years but I see no need for them to hit me, even more so when my 403s provide real people with how-to-contact info.

Thus about I echo Bill's comments in the July, 2008, thread linked above:

"I only allow Google IPs that are actually the Googlebot, or other plainly identified crawler from Google that I allow."

AppEngine anything is among those crawler/whatevers I don't. You?

[dstiles]

Unfortunately google runs bots on IPs that do not have a proper rDNS but are essential AND they use the same IP to run various user proxies and services.

Last night I had to verify a "base" aka "froogle" site. When I attempted it google hit an IP that had been 403'd for bad behaviour, probably a mis-behaved access through a transcoder proxy. Same applies to feedfetcher, which skips about all over the place. I currently have four such IPs blocked in the 74.125.16.0 - 74.125.16.255 range.

Google isn't alone in this but they have to be the leader. "We always have a proper rDNS for our bots". Yeah, right! :(

On topic: I have appengine-google blocked but no recent hits. I guess my sites are not worthy of attention from site-scrapers... sorry, I mean SEOs. :)

[Pfui]

Thanks for the reply, ds. One point of clarification:

That spammy Hosting site/SEO guy/downforeveryoneorjustme.com and /downforeveryone.com tie-in doesn't appear to be scraper-related.

Data-related, yep (incl. google-analytics.com and quantserve.com/quantcast.com). Beyond that, dunno. And I dun' like dunnos:)

[dstiles]

Why would an SEO hit someone else's web site? I would have thought it's to discover the contents of the site in order to do something similar or discover why the site ranks higher than his clients' - or even just to learn something. To do that it's necessary to pull a site and examine it.

Well, that's my theory. :)