Kindle & Amazon IPs?

[keyplyr]

So this is my 1st sighting of a Kindle user coming in with an Amazon IP non crawler, non AWS or at least the range isn't registered as AWS. User requested a popular web page and was stopped cold because the IP range is blocked; no other files requested. Could this be Fire? Or maybe Amazon is also an ISP in Ireland?


UA: Mozilla/5.0 (Linux; U; en-US) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) Version/4.0 Kindle/3.0 (screen 600x800; rotate)

Amazon (Ireland)
rDNS: no
87.238.80.0 - 87.238.81.255
87.238.80.0/21

[wilderness]

keyplr,
There are some occasional compliant users @ Amazon.

I recall Jim keeping one range open for a bot that was beneficial to his sites, although I don't recall the bot or range.

[incrediBILL]

Compliant Amazon users are kind of like Sasquatch sightings, people claim they see it but proof is elusive ;)

[wilderness]

Hey Bill,
I realize that "compliant" and "Amazon" are oxymoron's, however, and for the soothsayers, there's always exceptions.

[incrediBILL]

Yes, I'm aware, but I'm not sure the exceptions that play in the Amazon cesspool deserve a break. If you reward that behavior they'll never learn, but it doesn't seem like the few trying to teach them are making any headway.

[dstiles]

I'm with Bill on this one - Amazon-Anything stays blocked!

[keyplyr]

This intention of this thread was not so much about AWS, but Kindle's Fire.

The dynamic in question here is that Amazon's new Kindle Fire, which may become popular given the very low purchase price, purportedly requests our web pages via the Amazon cloud (EC2.)

It is yet-to-be-determined whether Amazon will try and cache (copy) our web pages or possibly just browse from EC2 proxy connections.

[Pfui]

I'm not sure we've spotted Fire's UA yet -- unless they didn't change anything... [webmasterworld.com...]

[Pfui]

Just found this specific info via the link below (bold emphasis mine):

Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-80) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

Source: "The User Agent String of Kindle Fire Revealed" [11-18-11] [labnol.org...]

[keyplyr]

Yes, same UA as previous discussion. The difference is where it came from, which presented the hypothesis.

::I feel so intelligent using words like that::

[lucy24]

I never made it past the IP range.

87.238.80.0 - 87.238.81.255
87.238.80.0/21

Shouldn't that be /23? Or possibly ..87.?

You know you've been locking out too many robots when you no longer have to spend hours with an abacus to figure these things out. :(

[keyplyr]

Good eye lucy24 - Yes, 87.238.80.0 - 87.238.81.255 does resolve to 87.238.80.0/23

The 87.238.80.0/21 came from DNSstuff and probably reflects route. I was using their old browser add-on tool. Probably should do them all manually for better specificity.

[edited by: keyplyr at 11:53 am (utc) on Nov 26, 2011]

[keyplyr]

Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Well that's simple enough.

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-80) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

[Pfui]

Some may need to tweak old UA blocks for: accelerate

(I did, re the now d/c'd Google Web Accelerator [webmasterworld.com...] and from way, way back to modem days.)

[Pfui]

This just in, emphasis mine. I still find it odd there's no indication of Kindle per se:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-84) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

Really Big FYI: Silk hit html files, graphics, and favicon from amazonaws.com AND, seconds later, from Comcast:

ec2-50-16-61-105.compute-1.amazonaws.com
06:41:07 /dir/filename.html
06:41:14 /favicon.ico
06:41:35 /dir/filename.html
ec2-184-73-52-100.compute-1.amazonaws.com
06:46:56 /dir/filename.html
ec2-204-236-204-113.compute-1.amazonaws.com
07:26:49 /dir/filename.html

c-98-227-60-2nn.hsd1.il.comcast.net
06:41:11 /dir/filename.html
06:41:15 /favicon.ico
06:41:40 /dir/filename.html
06:47:01 /dir/filename.html
07:26:53 /dir/filename.html

AWS Silk = 403 because of AWS [webmasterworld.com...] independent of AWS-cached acceleration... Or non-acceleration: One of many such reports: [foxnews.com...]

Comcast Silk = 200. No clue what the latter actually 'saw' because ironically, the files viewed were highly formatted to mimic hard copy and would lose a LOT in any translation.

[keyplyr]

Too much of a coincidence to be 2 users. So it does appear Silk is, at least in some way, operating from the Amazon Cloud.

Scenario: Silk equipped Kindle user who's ISP is Comcast finds our site. The tablet gets all its internet capabilities from the Amazon Mother Ship which hits our site first and displays it on the tablet for the Comcast user.

Question: were the requested files the same? I assume they were from the same web page, but was there a division in files types between the Amazon IP and the Comcast IP?

[Pfui]

The files requested were identical, even down to specific rewritten referrers (based on off-server requests for certain files, not the UAs doing the requesting).

For some reason I thought Kindle users only used/were limited to Amazon IPs from the get-go, so I was surprised to see the Comcast-based fellow traveler, let alone the totally resource-wasteful double-hits-double-hits. If they all tag-team, they're all getting rewritten, sorry.

Hmm. There appears to be the opposite setting out there: Silk-Accelerated=false

If/when I see that -- and it hits solo -- I may simply rewrite the =true types to a 'please switch gears' page. (But really, if you can enable acceleration, and They say it's faster, why wouldn't you? Who doesn't want to Go. Faster. Now!)

[keyplyr]

In a way, I like the fact that the user's ISP (in this case Comcast) is also requesting the same file. Since all Amazon IPs will be blocked at my server, hopefully the user still gets to see what they came for.

[Pfui]

1.) Another Silk twin hit thing to report, even including identical G search referrers (that'll make stats crazy!). Every file hit first by AWS, then two to four seconds later by Frontier, using:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-84) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

FIRST:

ec2-204-236-176-65.us-west-1.compute.amazonaws.com

SECONDS LATER:

50-103-7-1nn.dklb.il.frontiernet.net

FWIW, the identical G REF began:

http://www.google.com/url?sa=t&rct=j&q=[keyword=here]&source=web&cd=1&ved=[etc.]

(No UA apparent in REF; probably in lengthy alphanumeric string sections.)

2.) So okay, here's the thing. All non bot-related AWS hits are 403'd from the get-go, so technically, AWS shouldn't even 'know' there are graphics on pages it can't get to in the first place, right? But AWS does -- goes right for 'em:

AWS Silk:

1n:34:10 / 403
1n:34:17 /dir/exampleA.jpg 403
1n:34:17 /dir/exampleB.jpg 403
1n:34:17 /dir/exampleC.jpg 403
1n:34:17 /bots.jpg 200

FRONTIER Silk:

1n:34:12 / 200
1n:34:21 /dir/exampleA.jpg 200
1n:34:21 /dir/exampleB.jpg 200
1n:34:21 /dir/exampleC.jpg 200
[Didn't go for bots.jpg <= AWS hit was 200]

I've blocked amazonaws.com for two years. So there's NO way AWS could know file location details -- unless they're working off unauthorized caches/copies crawled by bare IPs or other Hosts. Dammit.

[dstiles]

Only one "silk" hit so far this month (a double one, AWS first)...

--------------------
Date: 04/Dec/2011 17:37:58
IP: 107.22.104.199 (Amazon_AWS)
Proxy Fwd-For IP: 68.9.227.187::HTTP/1.1 silk

UA: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Silk/1.1.0-84) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Silk-Accelerated=true

REFERER: [google.com...]

--------------------
Date: 04/Dec/2011 17:37:59
IP: 68.9.227.187 (Cox)

UA: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Silk/1.1.0-84) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 Silk-Accelerated=true

Referer: [google.com...]

--------------------

The fact that the referer was G's serps suggests there's an interaction going on.

[Pfui]

The fact that the referer was G's serps suggests there's an interaction going on.

You mean like a real person searched G for something, and clicked through to you? (That's what I saw; more or less typical G-as-REF behavior, except it was doubled.)

[dstiles]

To me it implies two services running in parallel - one from AWS and the other from the actual searcher (cox in my example).

This suggests something akin to a "toolbar" or "av-like" app running, one feeding back to AWS, the other doing what it's supposed to (ie, fetch a page). Timing may be disputable: if the ISP is slower than AWS then AWS could get its request in first (my example); if the AWS route was slower then vice versa.

So what is really happening? Is an AWS app, true to Cloud form, scraping data, perhaps for "popularity" testing? What relationship does it have to these browsers? What, in fact ARE the browsers? Is Silk a plug-in app or part of the browser? Is the world coming to an end? Who knows?! :(

[Pfui]

FWIW... [webmasterworld.com...]

[dstiles]

Thanks. I'd almost forgotten that - memory is not one of my good points. :)

Still doesn't explain the parallel-ishness. :(

If silk (as whatever browser it pretends to be) is running from within the cloud one would expect ONLY the cloud IP. Since it's coming at us with both cloud AND ISP IPs there has to be more to it, and my surmise is it's not going to be to webmasters' advantage.

A thought occurs: we all block AWS here, right? So: if AWS is merely acting as a pre-loader, when it reports to the browser that it can't get the page, perhaps that's when the browser tries to access it. Is there anyone here who can test that? If AWS is not blocked, does the broadband IP still come in?

[Pfui]

FWIW: I finally saw one of these, from an AT&T address by itself -- no Silk, no twin hits from AWS:

76.249.189.2nn
Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

[keyplyr]

Should increase quite a lot after Christmas.

[Pfui]

I've seen zilch increase. Am surprised.

Btw..

"Amazon Kindle Fire browser hacked for your Android pleasure" [theregister.co.uk...]

[dstiles]

"Silk offloads most of the rendering of web pages onto the cloud..."

Which explains some things we've discussed and adds another way of killing potential visitors to our web sites. :(