Amazon AWS Hosts Bad Bots

Continuation Thread

[incrediBILL]

This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.

[Key_Master]

Google's PageSpeed Insights is now using Amazon AWS to generate website performance reports. IPs vary.

User-Agent:

Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36 PTST/111

[Key_Master]

Here is the IP list. Notice the Verizon IP. All of these hits (plus the duplicates I deleted) was generated from one performance report.

50.18.90.254 (ec2-50-18-90-254.us-west-1.compute.amazonaws.com)
54.215.123.198 (ec2-54-215-123-198.us-west-1.compute.amazonaws.com)
54.215.90.1 (ec2-54-215-90-1.us-west-1.compute.amazonaws.com)
54.241.45.167 (ec2-54-241-45-167.us-west-1.compute.amazonaws.com)
72.66.115.10 (static-72-66-115-10.washdc.fios.verizon.net)
74.125.182.24 (74.125.182.24)
74.125.182.25 (74.125.182.25)
74.125.182.27 (74.125.182.27)
74.125.182.29 (74.125.182.29)
74.125.182.81 (74.125.182.81)
74.125.182.82 (74.125.182.82)
74.125.182.84 (74.125.182.84)
74.125.183.17 (74.125.183.17)
74.125.183.19 (74.125.183.19)
74.125.183.20 (74.125.183.20)
74.125.183.21 (74.125.183.21)
74.125.183.22 (74.125.183.22)
74.125.186.23 (74.125.186.23)
74.125.187.151 (74.125.187.151)

[Key_Master]

Additional info:

HTTP_X_FORWARDED_FOR{'54.241.45.167'}
HTTP_X_PSS_LOOP{'pagespeed_proxy'}

Sorry for the multiple posts. I have a lot of data to sort through.

::added::

Verizon details:
Agent:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)

Didn't execute JavaScript. Got 403'd.

Only the 74.125. range used the X_headers I posted above. Looks like this service still has some bugs:

Agent:

Mozilla/5.0 (en-US) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION] pss-webkit-request

These did not execute JavaScript either.

[dstiles]

New (to me) amazon IP range...

178.236.0.0 - 178.236.15.255

Found, oddly enough, by a report that amazon(.)in was now active. The Indian web site is apparently hosted in Ireland.

All MX records for the domain are in Amazon US ranges.

No indication of dates in DNS so no idea when the range was registered.

[keyplyr]

Good find, thanks!

[Dijkgraaf]

And in the future some of the bad bots coming to you from AWS might be from the CIA
[theregister.co.uk...]

[dstiles]

Maybe bots, but from commentaries this week it seems that the garnered data from such as verizon and others ("possibly" including facebook and G) is being processed by cloud-based computers.

Which, given the potential vulnerability and actual use by criminals of cloud-based services, is very scary!

[not2easy]

I bumped into that the other day too, but in a very slightly different range:
inetnum: 178.236.0.0 - 178.236.7.255
netname: IE-AMAZON
descr: Amazon Data Services Ireland
country: IE

An undesirable visitor showed up and it was just barely outside an OVH block so I ran a whois and got that info.

[lucy24]

Query: Is Merck simply selling off the 54. range piece by piece to Amazon?

Witness:

54.215.114.193 - - <snip> "GET /ebooks/perez/Perez.html HTTP/1.1" 200 11940 "http://yandex.ru/yandsearch?text=the+perez&lr=213" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" 
54.215.58.102 - - <snip> "GET /ebooks/perez/Perez.html HTTP/1.1" 200 11940 "http://yandex.ru/yandsearch?text=the+perez&lr=213" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

... in each case accompanied by a full complement of supporting files with plausible timing. If there hadn't been two of them, I would never have noticed.

To forestall the obvious rejoinder: MSIE 6 gets a free pass under certain circumstances. Once of those is an apparent search-engine query. I don't know what "lr=213" (that exact number) means, but I am now going to block it independently, because I have only seen it in forged yandex referers.

Further puttering around reveals that 54.222-223, which I'd somehow got flagged as Australia, is in fact Amazon China. Click! And still further puttering suggests that all of 54.192.0.0/10 now belongs to assorted tentacles of Amazon. Some of them look definitely humanoid, which is annoying.

[keyplyr]

Thanks Lucy, didn't have this range:

Amazon, China
54.222.0.0 - 54.223.255.255
54.222.0.0/15

[dstiles]

Lucy - thanks for the downward extension. I only had it down to 208 before now. I now have the range 54.192/10 blocked (ie everything from 192).

I now have 28 ranges for Amazon, all blocked, deserved or not.

I've recently been reviewing my proxy blocking policy: a lot of mobile devices are coming from proxies. Specifically I notice a lot of hits are using amazon IPs as proxies. They are out of luck but I think I'm going to check for proxy+amazon and show a warning page: "Do not use Amazon!".

[JohnG]

Latest hits from AWS, just in the last fifteen minutes:
54.226.186.253
184.72.180.14
54.205.132.55
54.211.123.193

[dstiles]

The upper /10 of 54 is all amazon. See an earlier posting for latest amazon ranges.

[webmasterworld.com...]

[webmasterworld.com...]

[keyplyr]

a lot of mobile devices are coming from proxies. Specifically I notice a lot of hits are using amazon IPs as proxies

I've noticed this as well. In the last couple months I've received a couple emails from different users asking why they're being blocked. Tracing it back, they were either tablet or mobile phones coming from Amazon ranges.

[lucy24]

Tracing it back, they were either tablet or mobile phones coming from Amazon ranges.

Are there distinct subranges where you can poke holes? Or do you detour to check for an X-Forwarded-For header? (Assuming there was one.)

[keyplyr]

I haven't been looking at the ranges long enough to determine whether they were assigned, or just an open proxy that moves dynamically as available. As we know, Amazon is not forthcoming with their customer info.

Now that you bring it up, I may add some conditions to get a better watch with headers.

[dstiles]

I block amazon, period. If I detect an IP being used for proxying I issue a warning** on the reject page to the effect they should use direct IP access. This also applies to some G, Y and other proxy sources.

** that's if I already know the forward-for IP is a valid DSL line. If it's a server they are on their own; if I do not yet know its status, they are on their own until I parse it later in the day.

[keyplyr]

@dstiles or anyone :)

Do you have more Amazom 54's than these? Thanks.

54.192.0.0/10
54.222.0.0/15
54.224.0.0/11
54.240.0.0/12

[feliceheaton]

I have a whole host of Amazon IPs that are driving me crazy, mostly FlipboardProxy taking down my website, but here's the 54s that I have...

54.196.10.155
54.196.18.56
54.196.46.109
54.196.48.57
54.196.57.188
54.204.67.160
54.204.72.120
54.204.77.56
54.204.87.22
54.204.97.228
54.204.140.221
54.204.162.23
54.204.169.245
54.204.183.36
54.204.201.211
54.204.210.84
54.204.218.62
54.204.226.35
54.204.231.206
54.204.252.17
54.204.254.21
54.205.65.127
54.205.112.94
54.205.130.157
54.205.165.112
54.205.174.156
54.205.183.208
54.205.197.62
54.211.7.251
54.211.35.243
54.211.87.61
54.211.122.104
54.211.178.177
54.211.181.186
54.211.241.122
54.221.7.29
54.221.178.53
54.224.12.23
54.224.84.102
54.224.138.61
54.224.145.47
54.224.215.202
54.224.248.191
54.224.251.198
54.225.2.51
54.225.46.14
54.225.52.196
54.226.88.148
54.226.92.192
54.226.107.132
54.226.193.197
54.226.218.236
54.226.248.245
54.227.14.52
54.227.57.201
54.227.69.253
54.227.92.13
54.227.160.53
54.227.220.217
54.234.8.141
54.234.31.143
54.234.70.248
54.234.137.246
54.234.179.144
54.234.183.175
54.237.57.156
54.242.8.97
54.242.70.1
54.242.88.238
54.242.249.251
54.243.9.45

In top of those, I have others. Mostly in these ranges:

23.20.
23.22.
23.23.
50.16.
50.17.
50.19.
67.202.
72.44.
75.101.
107.20.
107.22.
174.30.
174.129.
184.72.
204.236.

Like I said, the majority of these are flipboard. I also get hit by Amazon IPs using a "-" user agent string too, but those are auto-blocked by my htaccess.

I feel so special when they come by to make my server throw a 500 error.

[Angonasec]

KeyP:

I block two extra for AWS/NSA:
54.200.0.0/14
54.204.0.0/15

[Angonasec]

Welcome Felice, I see you're already addicted.

Neater to list as IP range AND cidr as your fellow log-watchers use various blocking filtering mechanisms.
About which we keep mum for obvious reasons.

[dstiles]

Keyplr: I block the complete amazon range at 54 as:

54.192.0.0 - 54.255.255.255
54.192.0.0/10

My current set of amazon ranges (Arin, Apnic, Ripe - all blocked) is...

8.18.144.0 - 8.18.145.255
23.20.0.0 - 23.23.255.255
27.0.0.0 - 27.0.3.255
46.51.128.0 - 46.51.255.255
46.137.0.0 - 46.137.255.255
50.16.0.0 - 50.19.255.255
50.112.0.0 - 50.112.255.255
54.192.0.0 - 54.255.255.255
67.202.0.0 - 67.202.63.255
72.21.192.0 - 72.21.223.255
72.44.32.0 - 72.44.63.255
75.101.128.0 - 75.101.255.255
79.125.0.0 - 79.125.127.255
87.238.80.0 - 87.238.87.255
103.4.8.0 - 103.4.15.255
107.20.0.0 - 107.23.255.255
122.248.192.0 - 122.248.255.255
174.129.0.0 - 174.129.255.255
175.41.128.0 - 175.41.255.255
176.32.64.0 - 176.32.127.255
176.34.0.0 - 176.34.255.255
177.71.128.0 - 177.71.255.255
178.236.0.0 - 178.236.15.255
184.72.0.0 - 184.73.255.255
184.169.128.0 - 184.169.255.255
199.255.192.0 - 199.255.195.255
204.236.128.0 - 204.236.255.255
205.251.192.0 - 205.251.255.255
207.171.160.0 - 207.171.191.255
216.182.224.0 - 216.182.239.255

[lucy24]

54.192.0.0/10
54.222.0.0/15
54.224.0.0/11
54.240.0.0/12

b, c and d are contained within a. d is contained within c

/10 = 192-255
/15 = 222-223
/11 = 224-255
/12 = 240-255

You internalize it after a while. Honest, Don, you do.

[keyplyr]

b, c and d are contained within a. d is contained within c

So you're saying 54.192.0.0/10 = 54.192.0.0 - 54.255.255.255?

I had thought 4.192.0.0/10 = 54.192.0.0 - 54.221.255.255

[edited by: keyplyr at 10:16 pm (utc) on Jan 4, 2014]

[wilderness]

You internalize it after a while. Honest, Don, you do.

No need to internalize anything (besides, all my internalization space is occupied by widgets, and all that remains is sheer obstinence)

19[2-9]|2[0-5][0-9]

[lucy24]

I had thought 54.192.0.0/10 = 54.192.0.0 - 54.221.255.255

Nothing is -221. Well, except 220-221, aka /15 /23 /31

/10 = /2 = /18 = /26 depending on which 0-255 block you're looking at

192 = 128+64 = binary 11000000
192/2 /10 /18 /26 = 11xxxxxx = 192-255
192/3 /11 /19 /27 = 110xxxxx = 192-223
192/4 /12 /20 /28 = 1100xxxx = 192-207
192/5 /13 /21 /29 = 11000xxx = 192-199

[dstiles]

If you are running linux then look at GIP calculator.

For windows there are online and downloadable calculators from simple to "Why is it so complicated?"

A simple check, which I display for convenience on my IP-blocks manager:

Breakpoints within a /16 (ie from nnn.nnn.0.0 to nnn.nnn.255.255)...

/18: 0 64 128 192 256
/19: 0 32 64 96 128 160 192 224 256
/20: 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256
/21: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
(/22 is 4 /24s and /23 is 2 /24s)

Breakpoints within a /8 (ie from nnn.0.0.0 to nnn.255.255.255)...

/10: 0 64 128 192 256
/11: 0 32 64 96 128 160 192 224 256
/12: 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256
/13: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
(/14 is 4 /16s and /15 is 2 /16s)

[keyplyr]

Just a heads-up: Inside of AWS range 54.192.0.0/10 is Nokia Express, a mobile carrier bringing human traffic.

54.236.252.0/22
54.236.252.0 - 54.236.255.255

[thetrasher]

Cloud-based Nokia Xpress Browser [en.wikipedia.org]

NetRange: 54.244.56.0 - 54.244.63.255
CIDR: 54.244.56.0/21
Name: AWS-XPRESSSERVICES1

NetRange: 54.236.252.0 - 54.236.255.255
CIDR: 54.236.252.0/22
Name: AWS-XPRESSSERVICES2

NetRange: 54.209.248.0 - 54.209.251.255
CIDR: 54.209.248.0/22
Name: AWS-XPRESSSERVICES3