homepage Welcome to WebmasterWorld Guest from 174.129.103.100
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Kindle & Amazon IPs?
keyplyr




msg:4390655
 12:10 pm on Nov 24, 2011 (gmt 0)


So this is my 1st sighting of a Kindle user coming in with an Amazon IP non crawler, non AWS or at least the range isn't registered as AWS. User requested a popular web page and was stopped cold because the IP range is blocked; no other files requested. Could this be Fire? Or maybe Amazon is also an ISP in Ireland?


UA: Mozilla/5.0 (Linux; U; en-US) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) Version/4.0 Kindle/3.0 (screen 600x800; rotate)

Amazon (Ireland)
rDNS: no
87.238.80.0 - 87.238.81.255
87.238.80.0/21

 

wilderness




msg:4390736
 4:56 pm on Nov 24, 2011 (gmt 0)

keyplr,
There are some occasional compliant users @ Amazon.

I recall Jim keeping one range open for a bot that was beneficial to his sites, although I don't recall the bot or range.

incrediBILL




msg:4390746
 5:07 pm on Nov 24, 2011 (gmt 0)

Compliant Amazon users are kind of like Sasquatch sightings, people claim they see it but proof is elusive ;)

wilderness




msg:4390762
 5:47 pm on Nov 24, 2011 (gmt 0)

Hey Bill,
I realize that "compliant" and "Amazon" are oxymoron's, however, and for the soothsayers, there's always exceptions.

incrediBILL




msg:4390763
 5:51 pm on Nov 24, 2011 (gmt 0)

Yes, I'm aware, but I'm not sure the exceptions that play in the Amazon cesspool deserve a break. If you reward that behavior they'll never learn, but it doesn't seem like the few trying to teach them are making any headway.

dstiles




msg:4390836
 9:59 pm on Nov 24, 2011 (gmt 0)

I'm with Bill on this one - Amazon-Anything stays blocked!

keyplyr




msg:4391240
 8:22 am on Nov 26, 2011 (gmt 0)

This intention of this thread was not so much about AWS, but Kindle's Fire.

The dynamic in question here is that Amazon's new Kindle Fire, which may become popular given the very low purchase price, purportedly requests our web pages via the Amazon cloud (EC2.)

It is yet-to-be-determined whether Amazon will try and cache (copy) our web pages or possibly just browse from EC2 proxy connections.

Pfui




msg:4391243
 9:47 am on Nov 26, 2011 (gmt 0)

I'm not sure we've spotted Fire's UA yet -- unless they didn't change anything... [webmasterworld.com...]

Pfui




msg:4391244
 10:13 am on Nov 26, 2011 (gmt 0)

Just found this specific info via the link below (bold emphasis mine):

Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-80) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

Source: "The User Agent String of Kindle Fire Revealed" [11-18-11] [labnol.org...]

keyplyr




msg:4391245
 10:17 am on Nov 26, 2011 (gmt 0)

Yes, same UA as previous discussion. The difference is where it came from, which presented the hypothesis.

::I feel so intelligent using words like that::

lucy24




msg:4391247
 10:52 am on Nov 26, 2011 (gmt 0)

I never made it past the IP range.

87.238.80.0 - 87.238.81.255
87.238.80.0/21


Shouldn't that be /23? Or possibly ..87.?

You know you've been locking out too many robots when you no longer have to spend hours with an abacus to figure these things out. :(

keyplyr




msg:4391251
 11:48 am on Nov 26, 2011 (gmt 0)

Good eye lucy24 - Yes, 87.238.80.0 - 87.238.81.255 does resolve to 87.238.80.0/23

The 87.238.80.0/21 came from DNSstuff and probably reflects route. I was using their old browser add-on tool. Probably should do them all manually for better specificity.

[edited by: keyplyr at 11:53 am (utc) on Nov 26, 2011]

keyplyr




msg:4391255
 11:52 am on Nov 26, 2011 (gmt 0)

Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Well that's simple enough.

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-80) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

Pfui




msg:4391321
 6:11 pm on Nov 26, 2011 (gmt 0)

Some may need to tweak old UA blocks for: accelerate

(I did, re the now d/c'd Google Web Accelerator [webmasterworld.com...] and from way, way back to modem days.)

Pfui




msg:4393869
 7:34 pm on Dec 3, 2011 (gmt 0)

This just in, emphasis mine. I still find it odd there's no indication of Kindle per se:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-84) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

Really Big FYI: Silk hit html files, graphics, and favicon from amazonaws.com AND, seconds later, from Comcast:

ec2-50-16-61-105.compute-1.amazonaws.com
06:41:07 /dir/filename.html
06:41:14 /favicon.ico
06:41:35 /dir/filename.html
ec2-184-73-52-100.compute-1.amazonaws.com
06:46:56 /dir/filename.html
ec2-204-236-204-113.compute-1.amazonaws.com
07:26:49 /dir/filename.html

c-98-227-60-2nn.hsd1.il.comcast.net
06:41:11 /dir/filename.html
06:41:15 /favicon.ico
06:41:40 /dir/filename.html
06:47:01 /dir/filename.html
07:26:53 /dir/filename.html

AWS Silk = 403 because of AWS [webmasterworld.com...] independent of AWS-cached acceleration... Or non-acceleration: One of many such reports: [foxnews.com...]

Comcast Silk = 200. No clue what the latter actually 'saw' because ironically, the files viewed were highly formatted to mimic hard copy and would lose a LOT in any translation.

keyplyr




msg:4393916
 10:35 pm on Dec 3, 2011 (gmt 0)

Too much of a coincidence to be 2 users. So it does appear Silk is, at least in some way, operating from the Amazon Cloud.

Scenario: Silk equipped Kindle user who's ISP is Comcast finds our site. The tablet gets all its internet capabilities from the Amazon Mother Ship which hits our site first and displays it on the tablet for the Comcast user.

Question: were the requested files the same? I assume they were from the same web page, but was there a division in files types between the Amazon IP and the Comcast IP?

Pfui




msg:4393950
 1:32 am on Dec 4, 2011 (gmt 0)

The files requested were identical, even down to specific rewritten referrers (based on off-server requests for certain files, not the UAs doing the requesting).

For some reason I thought Kindle users only used/were limited to Amazon IPs from the get-go, so I was surprised to see the Comcast-based fellow traveler, let alone the totally resource-wasteful double-hits-double-hits. If they all tag-team, they're all getting rewritten, sorry.

Hmm. There appears to be the opposite setting out there: Silk-Accelerated=false

If/when I see that -- and it hits solo -- I may simply rewrite the =true types to a 'please switch gears' page. (But really, if you can enable acceleration, and They say it's faster, why wouldn't you? Who doesn't want to Go. Faster. Now!)

keyplyr




msg:4393953
 1:48 am on Dec 4, 2011 (gmt 0)

In a way, I like the fact that the user's ISP (in this case Comcast) is also requesting the same file. Since all Amazon IPs will be blocked at my server, hopefully the user still gets to see what they came for.

Pfui




msg:4394999
 1:46 am on Dec 7, 2011 (gmt 0)

1.) Another Silk twin hit thing to report, even including identical G search referrers (that'll make stats crazy!). Every file hit first by AWS, then two to four seconds later by Frontier, using:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-84) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

FIRST:

ec2-204-236-176-65.us-west-1.compute.amazonaws.com

SECONDS LATER:

50-103-7-1nn.dklb.il.frontiernet.net

FWIW, the identical G REF began:

http://www.google.com/url?sa=t&rct=j&q=[keyword=here]&source=web&cd=1&ved=[etc.]

(No UA apparent in REF; probably in lengthy alphanumeric string sections.)

2.) So okay, here's the thing. All non bot-related AWS hits are 403'd from the get-go, so technically, AWS shouldn't even 'know' there are graphics on pages it can't get to in the first place, right? But AWS does -- goes right for 'em:

AWS Silk:

1n:34:10 / 403
1n:34:17 /dir/exampleA.jpg 403
1n:34:17 /dir/exampleB.jpg 403
1n:34:17 /dir/exampleC.jpg 403
1n:34:17 /bots.jpg 200

FRONTIER Silk:

1n:34:12 / 200
1n:34:21 /dir/exampleA.jpg 200
1n:34:21 /dir/exampleB.jpg 200
1n:34:21 /dir/exampleC.jpg 200
[Didn't go for bots.jpg <= AWS hit was 200]

I've blocked amazonaws.com for two years. So there's NO way AWS could know file location details -- unless they're working off unauthorized caches/copies crawled by bare IPs or other Hosts. Dammit.

dstiles




msg:4395375
 9:39 pm on Dec 7, 2011 (gmt 0)

Only one "silk" hit so far this month (a double one, AWS first)...

--------------------
Date: 04/Dec/2011 17:37:58
IP: 107.22.104.199 (Amazon_AWS)
Proxy Fwd-For IP: 68.9.227.187::HTTP/1.1 silk

UA: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Silk/1.1.0-84) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Silk-Accelerated=true

REFERER: [google.com...]

--------------------
Date: 04/Dec/2011 17:37:59
IP: 68.9.227.187 (Cox)

UA: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Silk/1.1.0-84) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Silk-Accelerated=true

Referer: [google.com...]

--------------------

The fact that the referer was G's serps suggests there's an interaction going on.

Pfui




msg:4395428
 12:15 am on Dec 8, 2011 (gmt 0)

The fact that the referer was G's serps suggests there's an interaction going on.


You mean like a real person searched G for something, and clicked through to you? (That's what I saw; more or less typical G-as-REF behavior, except it was doubled.)

dstiles




msg:4395737
 8:55 pm on Dec 8, 2011 (gmt 0)

To me it implies two services running in parallel - one from AWS and the other from the actual searcher (cox in my example).

This suggests something akin to a "toolbar" or "av-like" app running, one feeding back to AWS, the other doing what it's supposed to (ie, fetch a page). Timing may be disputable: if the ISP is slower than AWS then AWS could get its request in first (my example); if the AWS route was slower then vice versa.

So what is really happening? Is an AWS app, true to Cloud form, scraping data, perhaps for "popularity" testing? What relationship does it have to these browsers? What, in fact ARE the browsers? Is Silk a plug-in app or part of the browser? Is the world coming to an end? Who knows?! :(

Pfui




msg:4395772
 10:04 pm on Dec 8, 2011 (gmt 0)

FWIW... [webmasterworld.com...]

dstiles




msg:4395789
 10:50 pm on Dec 8, 2011 (gmt 0)

Thanks. I'd almost forgotten that - memory is not one of my good points. :)

Still doesn't explain the parallel-ishness. :(

If silk (as whatever browser it pretends to be) is running from within the cloud one would expect ONLY the cloud IP. Since it's coming at us with both cloud AND ISP IPs there has to be more to it, and my surmise is it's not going to be to webmasters' advantage.

A thought occurs: we all block AWS here, right? So: if AWS is merely acting as a pre-loader, when it reports to the browser that it can't get the page, perhaps that's when the browser tries to access it. Is there anyone here who can test that? If AWS is not blocked, does the broadband IP still come in?

Pfui




msg:4400224
 6:17 pm on Dec 21, 2011 (gmt 0)

FWIW: I finally saw one of these, from an AT&T address by itself -- no Silk, no twin hits from AWS:

76.249.189.2nn
Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

keyplyr




msg:4400282
 10:00 pm on Dec 21, 2011 (gmt 0)

Should increase quite a lot after Christmas.

Pfui




msg:4404160
 8:00 pm on Jan 5, 2012 (gmt 0)

I've seen zilch increase. Am surprised.

Btw..

"Amazon Kindle Fire browser hacked for your Android pleasure" [theregister.co.uk...]

dstiles




msg:4404173
 8:45 pm on Jan 5, 2012 (gmt 0)

"Silk offloads most of the rendering of web pages onto the cloud..."

Which explains some things we've discussed and adds another way of killing potential visitors to our web sites. :(

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved