homepage Welcome to WebmasterWorld Guest from 54.204.249.184
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
AppEngine-Google; ( http://code.google.com/appengine; appid: mapthisli
GaryK




msg:4033693
 10:07 pm on Nov 29, 2009 (gmt 0)

AppEngine-Google; ( [code.google.com...] appid: mapthislink)
64.233.172.nn
No PTR
-----
OrgName: Google Inc.
OrgID: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
NetRange: 64.233.160.0 - 64.233.191.255
-----
READ ROBOTS.TXT? No
OBEYED ROBOTS.TXT? No
-----
We've discussed Google's AppEngine before, but this appears to be a new user agent.

I've decided to ban anything with AppEngine-Google in it as none of them has ever provided anything useful in exchange for access to my sites.

 

Pfui




msg:4036593
 12:59 am on Dec 4, 2009 (gmt 0)

It's making the rounds. From earlier today; note change in UA string (...unless your copy-paste stripped out the leading +, Gary?):

64.233.172.18
AppEngine-Google; (+http://code.google.com/appengine; appid: mapthislink)

robots.txt? NO

(The file requested was a recent Twitter mention. But thus far, "mapthislink" doesn't appear to be Twitter-specific.)

Pfui




msg:4037204
 7:39 pm on Dec 4, 2009 (gmt 0)

Two more hits today simultaneous with swarms of Twitter-related UAs confirm mapthislink's connection.

FWIW, I've yet to see any ill effect from blocking pretty much ALL Twitter fellow travelers for one reason or another, e.g., amazonaws.com-hosted; no-robots.txt; HEAD requests; bad or no UAs; ad nauseam hits, etc.

GaryK




msg:4037291
 9:32 pm on Dec 4, 2009 (gmt 0)

unless your copy-paste stripped out the leading +

I coded my log analysis software to strip-out plus signs. From what I've seen all they do is hard-code spaces into an otherwise space-delimited log file,

Receptional Andy




msg:4037314
 10:19 pm on Dec 4, 2009 (gmt 0)

I've decided to ban anything with AppEngine-Google in it as none of them has ever provided anything useful in exchange for access to my sites.

I have the same conclusion. This method seems to be faddish at the moment amongst those who enjoy the novelty of masquerading as "Google approved" via a Google IP range.

jabz




msg:4037454
 2:26 am on Dec 5, 2009 (gmt 0)

@GaryK do you use Google's OpenID service or anything like that on your website?

Pfui




msg:4037571
 7:43 am on Dec 5, 2009 (gmt 0)

@Gary: When UAs you report are in any way altered by your software, might it be possible for you to please also/instead post the original, actual strings? It's really difficult to confidently block on specifics otherwise.

For example, the UA in the OP would automatically get caught by a no-space-after-left-paren rule. But because a key bit is missing, replaced by a space, the rule wouldn't kick in and absent other rules, the bot would get in. So anyway, TIA if you can include unaltered UAs:)

GaryK




msg:4037749
 5:18 pm on Dec 5, 2009 (gmt 0)

Nope none of my sites uses Google OpenID.

Pfui, that's no problem at all. Here's the raw UA for this one:

AppEngine-Google;+(+http://code.google.com/appengine;+appid:+mapthislink)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved