Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Amazon AWS Hosts Bad Bots

Continuation Thread

     
11:16 pm on May 16, 2013 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14622
votes: 88


This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.
2:05 am on May 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


Google's PageSpeed Insights is now using Amazon AWS to generate website performance reports. IPs vary.

User-Agent:
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36 PTST/111
2:14 am on May 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


Here is the IP list. Notice the Verizon IP. All of these hits (plus the duplicates I deleted) was generated from one performance report.

50.18.90.254 (ec2-50-18-90-254.us-west-1.compute.amazonaws.com)
54.215.123.198 (ec2-54-215-123-198.us-west-1.compute.amazonaws.com)
54.215.90.1 (ec2-54-215-90-1.us-west-1.compute.amazonaws.com)
54.241.45.167 (ec2-54-241-45-167.us-west-1.compute.amazonaws.com)
72.66.115.10 (static-72-66-115-10.washdc.fios.verizon.net)
74.125.182.24 (74.125.182.24)
74.125.182.25 (74.125.182.25)
74.125.182.27 (74.125.182.27)
74.125.182.29 (74.125.182.29)
74.125.182.81 (74.125.182.81)
74.125.182.82 (74.125.182.82)
74.125.182.84 (74.125.182.84)
74.125.183.17 (74.125.183.17)
74.125.183.19 (74.125.183.19)
74.125.183.20 (74.125.183.20)
74.125.183.21 (74.125.183.21)
74.125.183.22 (74.125.183.22)
74.125.186.23 (74.125.186.23)
74.125.187.151 (74.125.187.151)
2:26 am on May 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


Additional info:

HTTP_X_FORWARDED_FOR{'54.241.45.167'}
HTTP_X_PSS_LOOP{'pagespeed_proxy'}


Sorry for the multiple posts. I have a lot of data to sort through.

::added::

Verizon details:
Agent:
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)


Didn't execute JavaScript. Got 403'd.

Only the 74.125. range used the X_headers I posted above. Looks like this service still has some bugs:

Agent:
Mozilla/5.0 (en-US) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION] pss-webkit-request


These did not execute JavaScript either.
7:03 pm on June 5, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


New (to me) amazon IP range...

178.236.0.0 - 178.236.15.255

Found, oddly enough, by a report that amazon(.)in was now active. The Indian web site is apparently hosted in Ireland.

All MX records for the domain are in Amazon US ranges.

No indication of dates in DNS so no idea when the range was registered.
7:08 pm on June 5, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64



Good find, thanks!
2:40 am on June 10, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
posts:1108
votes: 0


And in the future some of the bad bots coming to you from AWS might be from the CIA
[theregister.co.uk...]
6:48 pm on June 10, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


Maybe bots, but from commentaries this week it seems that the garnered data from such as verizon and others ("possibly" including facebook and G) is being processed by cloud-based computers.

Which, given the potential vulnerability and actual use by criminals of cloud-based services, is very scary!
2:05 am on June 11, 2013 (gmt 0)

Moderator from US 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2563
votes: 48


I bumped into that the other day too, but in a very slightly different range:
inetnum: 178.236.0.0 - 178.236.7.255
netname: IE-AMAZON
descr: Amazon Data Services Ireland
country: IE

An undesirable visitor showed up and it was just barely outside an OVH block so I ran a whois and got that info.
10:53 pm on July 17, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12702
votes: 244


Query: Is Merck simply selling off the 54. range piece by piece to Amazon?

Witness:
54.215.114.193 - - <snip> "GET /ebooks/perez/Perez.html HTTP/1.1" 200 11940 "http://yandex.ru/yandsearch?text=the+perez&lr=213" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" 
54.215.58.102 - - <snip> "GET /ebooks/perez/Perez.html HTTP/1.1" 200 11940 "http://yandex.ru/yandsearch?text=the+perez&lr=213" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

... in each case accompanied by a full complement of supporting files with plausible timing. If there hadn't been two of them, I would never have noticed.

To forestall the obvious rejoinder: MSIE 6 gets a free pass under certain circumstances. Once of those is an apparent search-engine query. I don't know what "lr=213" (that exact number) means, but I am now going to block it independently, because I have only seen it in forged yandex referers.

Further puttering around reveals that 54.222-223, which I'd somehow got flagged as Australia, is in fact Amazon China. Click! And still further puttering suggests that all of 54.192.0.0/10 now belongs to assorted tentacles of Amazon. Some of them look definitely humanoid, which is annoying.
7:32 pm on July 18, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64




Thanks Lucy, didn't have this range:

Amazon, China
54.222.0.0 - 54.223.255.255
54.222.0.0/15
8:25 pm on July 18, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


Lucy - thanks for the downward extension. I only had it down to 208 before now. I now have the range 54.192/10 blocked (ie everything from 192).

I now have 28 ranges for Amazon, all blocked, deserved or not.

I've recently been reviewing my proxy blocking policy: a lot of mobile devices are coming from proxies. Specifically I notice a lot of hits are using amazon IPs as proxies. They are out of luck but I think I'm going to check for proxy+amazon and show a warning page: "Do not use Amazon!".
6:46 am on Oct 28, 2013 (gmt 0)

New User

5+ Year Member

joined:Oct 15, 2009
posts:2
votes: 0


Latest hits from AWS, just in the last fifteen minutes:
54.226.186.253
184.72.180.14
54.205.132.55
54.211.123.193
8:25 pm on Oct 28, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


The upper /10 of 54 is all amazon. See an earlier posting for latest amazon ranges.

[webmasterworld.com...]

[webmasterworld.com...]
9:47 pm on Oct 28, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64



a lot of mobile devices are coming from proxies. Specifically I notice a lot of hits are using amazon IPs as proxies

I've noticed this as well. In the last couple months I've received a couple emails from different users asking why they're being blocked. Tracing it back, they were either tablet or mobile phones coming from Amazon ranges.
10:08 pm on Oct 28, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12702
votes: 244


Tracing it back, they were either tablet or mobile phones coming from Amazon ranges.

Are there distinct subranges where you can poke holes? Or do you detour to check for an X-Forwarded-For header? (Assuming there was one.)
12:35 am on Oct 29, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64


I haven't been looking at the ranges long enough to determine whether they were assigned, or just an open proxy that moves dynamically as available. As we know, Amazon is not forthcoming with their customer info.

Now that you bring it up, I may add some conditions to get a better watch with headers.
7:51 pm on Oct 29, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


I block amazon, period. If I detect an IP being used for proxying I issue a warning** on the reject page to the effect they should use direct IP access. This also applies to some G, Y and other proxy sources.

** that's if I already know the forward-for IP is a valid DSL line. If it's a server they are on their own; if I do not yet know its status, they are on their own until I parse it later in the day.
9:01 am on Jan 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64



@dstiles or anyone :)

Do you have more Amazom 54's than these? Thanks.

54.192.0.0/10
54.222.0.0/15
54.224.0.0/11
54.240.0.0/12
10:56 am on Jan 4, 2014 (gmt 0)

New User

joined:Jan 4, 2014
posts:1
votes: 0


I have a whole host of Amazon IPs that are driving me crazy, mostly FlipboardProxy taking down my website, but here's the 54s that I have...

54.196.10.155
54.196.18.56
54.196.46.109
54.196.48.57
54.196.57.188
54.204.67.160
54.204.72.120
54.204.77.56
54.204.87.22
54.204.97.228
54.204.140.221
54.204.162.23
54.204.169.245
54.204.183.36
54.204.201.211
54.204.210.84
54.204.218.62
54.204.226.35
54.204.231.206
54.204.252.17
54.204.254.21
54.205.65.127
54.205.112.94
54.205.130.157
54.205.165.112
54.205.174.156
54.205.183.208
54.205.197.62
54.211.7.251
54.211.35.243
54.211.87.61
54.211.122.104
54.211.178.177
54.211.181.186
54.211.241.122
54.221.7.29
54.221.178.53
54.224.12.23
54.224.84.102
54.224.138.61
54.224.145.47
54.224.215.202
54.224.248.191
54.224.251.198
54.225.2.51
54.225.46.14
54.225.52.196
54.226.88.148
54.226.92.192
54.226.107.132
54.226.193.197
54.226.218.236
54.226.248.245
54.227.14.52
54.227.57.201
54.227.69.253
54.227.92.13
54.227.160.53
54.227.220.217
54.234.8.141
54.234.31.143
54.234.70.248
54.234.137.246
54.234.179.144
54.234.183.175
54.237.57.156
54.242.8.97
54.242.70.1
54.242.88.238
54.242.249.251
54.243.9.45

In top of those, I have others. Mostly in these ranges:

23.20.
23.22.
23.23.
50.16.
50.17.
50.19.
67.202.
72.44.
75.101.
107.20.
107.22.
174.30.
174.129.
184.72.
204.236.

Like I said, the majority of these are flipboard. I also get hit by Amazon IPs using a "-" user agent string too, but those are auto-blocked by my htaccess.

I feel so special when they come by to make my server throw a 500 error.
2:28 pm on Jan 4, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


KeyP:

I block two extra for AWS/NSA:
54.200.0.0/14
54.204.0.0/15
2:31 pm on Jan 4, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


Welcome Felice, I see you're already addicted.

Neater to list as IP range AND cidr as your fellow log-watchers use various blocking filtering mechanisms.
About which we keep mum for obvious reasons.
8:16 pm on Jan 4, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


Keyplr: I block the complete amazon range at 54 as:

54.192.0.0 - 54.255.255.255
54.192.0.0/10

My current set of amazon ranges (Arin, Apnic, Ripe - all blocked) is...

8.18.144.0 - 8.18.145.255
23.20.0.0 - 23.23.255.255
27.0.0.0 - 27.0.3.255
46.51.128.0 - 46.51.255.255
46.137.0.0 - 46.137.255.255
50.16.0.0 - 50.19.255.255
50.112.0.0 - 50.112.255.255
54.192.0.0 - 54.255.255.255
67.202.0.0 - 67.202.63.255
72.21.192.0 - 72.21.223.255
72.44.32.0 - 72.44.63.255
75.101.128.0 - 75.101.255.255
79.125.0.0 - 79.125.127.255
87.238.80.0 - 87.238.87.255
103.4.8.0 - 103.4.15.255
107.20.0.0 - 107.23.255.255
122.248.192.0 - 122.248.255.255
174.129.0.0 - 174.129.255.255
175.41.128.0 - 175.41.255.255
176.32.64.0 - 176.32.127.255
176.34.0.0 - 176.34.255.255
177.71.128.0 - 177.71.255.255
178.236.0.0 - 178.236.15.255
184.72.0.0 - 184.73.255.255
184.169.128.0 - 184.169.255.255
199.255.192.0 - 199.255.195.255
204.236.128.0 - 204.236.255.255
205.251.192.0 - 205.251.255.255
207.171.160.0 - 207.171.191.255
216.182.224.0 - 216.182.239.255
8:33 pm on Jan 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12702
votes: 244


54.192.0.0/10
54.222.0.0/15
54.224.0.0/11
54.240.0.0/12

b, c and d are contained within a. d is contained within c

/10 = 192-255
/15 = 222-223
/11 = 224-255
/12 = 240-255

You internalize it after a while. Honest, Don, you do.
10:11 pm on Jan 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64



b, c and d are contained within a. d is contained within c


So you're saying 54.192.0.0/10 = 54.192.0.0 - 54.255.255.255?

I had thought 4.192.0.0/10 = 54.192.0.0 - 54.221.255.255

[edited by: keyplyr at 10:16 pm (utc) on Jan 4, 2014]

10:22 pm on Jan 4, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5408
votes: 2


You internalize it after a while. Honest, Don, you do.


No need to internalize anything (besides, all my internalization space is occupied by widgets, and all that remains is sheer obstinence)

19[2-9]|2[0-5][0-9]
11:11 pm on Jan 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12702
votes: 244


I had thought 54.192.0.0/10 = 54.192.0.0 - 54.221.255.255

Nothing is -221. Well, except 220-221, aka /15 /23 /31

/10 = /2 = /18 = /26 depending on which 0-255 block you're looking at

192 = 128+64 = binary 11000000
192/2 /10 /18 /26 = 11xxxxxx = 192-255
192/3 /11 /19 /27 = 110xxxxx = 192-223
192/4 /12 /20 /28 = 1100xxxx = 192-207
192/5 /13 /21 /29 = 11000xxx = 192-199
8:54 pm on Jan 5, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


If you are running linux then look at GIP calculator.

For windows there are online and downloadable calculators from simple to "Why is it so complicated?"

A simple check, which I display for convenience on my IP-blocks manager:

Breakpoints within a /16 (ie from nnn.nnn.0.0 to nnn.nnn.255.255)...

/18: 0 64 128 192 256
/19: 0 32 64 96 128 160 192 224 256
/20: 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256
/21: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
(/22 is 4 /24s and /23 is 2 /24s)

Breakpoints within a /8 (ie from nnn.0.0.0 to nnn.255.255.255)...

/10: 0 64 128 192 256
/11: 0 32 64 96 128 160 192 224 256
/12: 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256
/13: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
(/14 is 4 /16s and /15 is 2 /16s)
7:33 am on Jan 7, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5805
votes: 64





Just a heads-up: Inside of AWS range 54.192.0.0/10 is Nokia Express, a mobile carrier bringing human traffic.

54.236.252.0/22
54.236.252.0 - 54.236.255.255
2:31 pm on Jan 7, 2014 (gmt 0)

Junior Member

10+ Year Member

joined:June 25, 2005
posts:179
votes: 1


Cloud-based Nokia Xpress Browser [en.wikipedia.org]

NetRange: 54.244.56.0 - 54.244.63.255
CIDR: 54.244.56.0/21
Name: AWS-XPRESSSERVICES1

NetRange: 54.236.252.0 - 54.236.255.255
CIDR: 54.236.252.0/22
Name: AWS-XPRESSSERVICES2

NetRange: 54.209.248.0 - 54.209.251.255
CIDR: 54.209.248.0/22
Name: AWS-XPRESSSERVICES3
This 108 message thread spans 4 pages: 108