Forum Moderators: open
User-Agent:
AppEngine-Google; (+http://code.google.com/appengine)
Referer:
[fetchserver3.appspot.com...]
Headers:
Browser-like.
Four hits on same site on two pages, page A, page B, page A, page B (Page A has a map on it but may be coincidence). Got a 403 each time and gave up.
Attempt to view the referer page (via Sam Spade) returns a 404 with "Requested URL aaa not found on server" where aaa should (but doesn't) contain the url of the page.
The URL in the UA returns a 302 page, empty except for "The document has moved here" where "here" is /appengine/ which contains info on the app engine, beginning:
"Run your web applications on Google's infrastructure. Build apps on the same scalable systems that power Google applications."
Further information looks as if it's a hosting operation intended to oust real hosting services - at least the low-end ones. Whether google will give priority to these sites in their index...
Question: Why would one of my sites get hits from this tool if it's a site-builder? Guess answer: it's populating a site with my data?
So far it's getting blocked. Be interesting to see if more hits turn up.
edited by wilderness.
Twelve days after your heads up.
74.125.16.zz - - [11/Feb/2008:04:13:17 -0600] "GET /MyFolder/ HTTP/1.1" 200 13034 "WidgetSite" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
71.43.82.zzz - - [11/Feb/2008:04:13:52 -0600] "GET /SameFolder/ HTTP/1.1" 200 13034 "WidgetSite" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
This time the referer domain appears to be owned by google and is an old one. Last time it was owned by someone in San Francisco and the domain was new.
Anyone know why it's hitting my site? Am I being paranoid about scraping by a google tool?
"I run tests on domains to make sure that they are working well."
"No thanks. eat this 403!"
64.233.172.6 - - [13/Jun/2009:18:14:16 +0100] "GET / HTTP/1.1" 403 - "http://www.example.com/" "AppEngine-Google; (+http://code.google.com/appengine)"
