homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 44 message thread spans 2 pages: 44 ( [1] 2 > >     
Amazon AWS Hosts Bad Bots
Continuation Thread
incrediBILL




msg:4574829
 11:16 pm on May 16, 2013 (gmt 0)

This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.

 

Key_Master




msg:4576763
 2:05 am on May 23, 2013 (gmt 0)

Google's PageSpeed Insights is now using Amazon AWS to generate website performance reports. IPs vary.

User-Agent:
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36 PTST/111
Key_Master




msg:4576768
 2:14 am on May 23, 2013 (gmt 0)

Here is the IP list. Notice the Verizon IP. All of these hits (plus the duplicates I deleted) was generated from one performance report.

50.18.90.254 (ec2-50-18-90-254.us-west-1.compute.amazonaws.com)
54.215.123.198 (ec2-54-215-123-198.us-west-1.compute.amazonaws.com)
54.215.90.1 (ec2-54-215-90-1.us-west-1.compute.amazonaws.com)
54.241.45.167 (ec2-54-241-45-167.us-west-1.compute.amazonaws.com)
72.66.115.10 (static-72-66-115-10.washdc.fios.verizon.net)
74.125.182.24 (74.125.182.24)
74.125.182.25 (74.125.182.25)
74.125.182.27 (74.125.182.27)
74.125.182.29 (74.125.182.29)
74.125.182.81 (74.125.182.81)
74.125.182.82 (74.125.182.82)
74.125.182.84 (74.125.182.84)
74.125.183.17 (74.125.183.17)
74.125.183.19 (74.125.183.19)
74.125.183.20 (74.125.183.20)
74.125.183.21 (74.125.183.21)
74.125.183.22 (74.125.183.22)
74.125.186.23 (74.125.186.23)
74.125.187.151 (74.125.187.151)

Key_Master




msg:4576775
 2:26 am on May 23, 2013 (gmt 0)

Additional info:

HTTP_X_FORWARDED_FOR{'54.241.45.167'}
HTTP_X_PSS_LOOP{'pagespeed_proxy'}


Sorry for the multiple posts. I have a lot of data to sort through.

::added::

Verizon details:
Agent:
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)

Didn't execute JavaScript. Got 403'd.

Only the 74.125. range used the X_headers I posted above. Looks like this service still has some bugs:

Agent:
Mozilla/5.0 (en-US) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION] pss-webkit-request

These did not execute JavaScript either.

dstiles




msg:4581409
 7:03 pm on Jun 5, 2013 (gmt 0)

New (to me) amazon IP range...

178.236.0.0 - 178.236.15.255

Found, oddly enough, by a report that amazon(.)in was now active. The Indian web site is apparently hosted in Ireland.

All MX records for the domain are in Amazon US ranges.

No indication of dates in DNS so no idea when the range was registered.

keyplyr




msg:4581413
 7:08 pm on Jun 5, 2013 (gmt 0)


Good find, thanks!

Dijkgraaf




msg:4582651
 2:40 am on Jun 10, 2013 (gmt 0)

And in the future some of the bad bots coming to you from AWS might be from the CIA
[theregister.co.uk...]

dstiles




msg:4582888
 6:48 pm on Jun 10, 2013 (gmt 0)

Maybe bots, but from commentaries this week it seems that the garnered data from such as verizon and others ("possibly" including facebook and G) is being processed by cloud-based computers.

Which, given the potential vulnerability and actual use by criminals of cloud-based services, is very scary!

not2easy




msg:4582982
 2:05 am on Jun 11, 2013 (gmt 0)

I bumped into that the other day too, but in a very slightly different range:
inetnum: 178.236.0.0 - 178.236.7.255
netname: IE-AMAZON
descr: Amazon Data Services Ireland
country: IE

An undesirable visitor showed up and it was just barely outside an OVH block so I ran a whois and got that info.

lucy24




msg:4594023
 10:53 pm on Jul 17, 2013 (gmt 0)

Query: Is Merck simply selling off the 54. range piece by piece to Amazon?

Witness:
54.215.114.193 - - <snip> "GET /ebooks/perez/Perez.html HTTP/1.1" 200 11940 "http://yandex.ru/yandsearch?text=the+perez&lr=213" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
54.215.58.102 - - <snip> "GET /ebooks/perez/Perez.html HTTP/1.1" 200 11940 "http://yandex.ru/yandsearch?text=the+perez&lr=213" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

... in each case accompanied by a full complement of supporting files with plausible timing. If there hadn't been two of them, I would never have noticed.

To forestall the obvious rejoinder: MSIE 6 gets a free pass under certain circumstances. Once of those is an apparent search-engine query. I don't know what "lr=213" (that exact number) means, but I am now going to block it independently, because I have only seen it in forged yandex referers.

Further puttering around reveals that 54.222-223, which I'd somehow got flagged as Australia, is in fact Amazon China. Click! And still further puttering suggests that all of 54.192.0.0/10 now belongs to assorted tentacles of Amazon. Some of them look definitely humanoid, which is annoying.

keyplyr




msg:4594336
 7:32 pm on Jul 18, 2013 (gmt 0)



Thanks Lucy, didn't have this range:

Amazon, China
54.222.0.0 - 54.223.255.255
54.222.0.0/15

dstiles




msg:4594348
 8:25 pm on Jul 18, 2013 (gmt 0)

Lucy - thanks for the downward extension. I only had it down to 208 before now. I now have the range 54.192/10 blocked (ie everything from 192).

I now have 28 ranges for Amazon, all blocked, deserved or not.

I've recently been reviewing my proxy blocking policy: a lot of mobile devices are coming from proxies. Specifically I notice a lot of hits are using amazon IPs as proxies. They are out of luck but I think I'm going to check for proxy+amazon and show a warning page: "Do not use Amazon!".

JohnG




msg:4619464
 6:46 am on Oct 28, 2013 (gmt 0)

Latest hits from AWS, just in the last fifteen minutes:
54.226.186.253
184.72.180.14
54.205.132.55
54.211.123.193

dstiles




msg:4619630
 8:25 pm on Oct 28, 2013 (gmt 0)

The upper /10 of 54 is all amazon. See an earlier posting for latest amazon ranges.

[webmasterworld.com...]

[webmasterworld.com...]

keyplyr




msg:4619654
 9:47 pm on Oct 28, 2013 (gmt 0)


a lot of mobile devices are coming from proxies. Specifically I notice a lot of hits are using amazon IPs as proxies

I've noticed this as well. In the last couple months I've received a couple emails from different users asking why they're being blocked. Tracing it back, they were either tablet or mobile phones coming from Amazon ranges.

lucy24




msg:4619661
 10:08 pm on Oct 28, 2013 (gmt 0)

Tracing it back, they were either tablet or mobile phones coming from Amazon ranges.

Are there distinct subranges where you can poke holes? Or do you detour to check for an X-Forwarded-For header? (Assuming there was one.)

keyplyr




msg:4619694
 12:35 am on Oct 29, 2013 (gmt 0)

I haven't been looking at the ranges long enough to determine whether they were assigned, or just an open proxy that moves dynamically as available. As we know, Amazon is not forthcoming with their customer info.

Now that you bring it up, I may add some conditions to get a better watch with headers.

dstiles




msg:4619870
 7:51 pm on Oct 29, 2013 (gmt 0)

I block amazon, period. If I detect an IP being used for proxying I issue a warning** on the reject page to the effect they should use direct IP access. This also applies to some G, Y and other proxy sources.

** that's if I already know the forward-for IP is a valid DSL line. If it's a server they are on their own; if I do not yet know its status, they are on their own until I parse it later in the day.

keyplyr




msg:4635031
 9:01 am on Jan 4, 2014 (gmt 0)


@dstiles or anyone :)

Do you have more Amazom 54's than these? Thanks.

54.192.0.0/10
54.222.0.0/15
54.224.0.0/11
54.240.0.0/12

feliceheaton




msg:4635038
 10:56 am on Jan 4, 2014 (gmt 0)

I have a whole host of Amazon IPs that are driving me crazy, mostly FlipboardProxy taking down my website, but here's the 54s that I have...

54.196.10.155
54.196.18.56
54.196.46.109
54.196.48.57
54.196.57.188
54.204.67.160
54.204.72.120
54.204.77.56
54.204.87.22
54.204.97.228
54.204.140.221
54.204.162.23
54.204.169.245
54.204.183.36
54.204.201.211
54.204.210.84
54.204.218.62
54.204.226.35
54.204.231.206
54.204.252.17
54.204.254.21
54.205.65.127
54.205.112.94
54.205.130.157
54.205.165.112
54.205.174.156
54.205.183.208
54.205.197.62
54.211.7.251
54.211.35.243
54.211.87.61
54.211.122.104
54.211.178.177
54.211.181.186
54.211.241.122
54.221.7.29
54.221.178.53
54.224.12.23
54.224.84.102
54.224.138.61
54.224.145.47
54.224.215.202
54.224.248.191
54.224.251.198
54.225.2.51
54.225.46.14
54.225.52.196
54.226.88.148
54.226.92.192
54.226.107.132
54.226.193.197
54.226.218.236
54.226.248.245
54.227.14.52
54.227.57.201
54.227.69.253
54.227.92.13
54.227.160.53
54.227.220.217
54.234.8.141
54.234.31.143
54.234.70.248
54.234.137.246
54.234.179.144
54.234.183.175
54.237.57.156
54.242.8.97
54.242.70.1
54.242.88.238
54.242.249.251
54.243.9.45

In top of those, I have others. Mostly in these ranges:

23.20.
23.22.
23.23.
50.16.
50.17.
50.19.
67.202.
72.44.
75.101.
107.20.
107.22.
174.30.
174.129.
184.72.
204.236.

Like I said, the majority of these are flipboard. I also get hit by Amazon IPs using a "-" user agent string too, but those are auto-blocked by my htaccess.

I feel so special when they come by to make my server throw a 500 error.

Angonasec




msg:4635051
 2:28 pm on Jan 4, 2014 (gmt 0)

KeyP:

I block two extra for AWS/NSA:
54.200.0.0/14
54.204.0.0/15

Angonasec




msg:4635052
 2:31 pm on Jan 4, 2014 (gmt 0)

Welcome Felice, I see you're already addicted.

Neater to list as IP range AND cidr as your fellow log-watchers use various blocking filtering mechanisms.
About which we keep mum for obvious reasons.

dstiles




msg:4635099
 8:16 pm on Jan 4, 2014 (gmt 0)

Keyplr: I block the complete amazon range at 54 as:

54.192.0.0 - 54.255.255.255
54.192.0.0/10

My current set of amazon ranges (Arin, Apnic, Ripe - all blocked) is...

8.18.144.0 - 8.18.145.255
23.20.0.0 - 23.23.255.255
27.0.0.0 - 27.0.3.255
46.51.128.0 - 46.51.255.255
46.137.0.0 - 46.137.255.255
50.16.0.0 - 50.19.255.255
50.112.0.0 - 50.112.255.255
54.192.0.0 - 54.255.255.255
67.202.0.0 - 67.202.63.255
72.21.192.0 - 72.21.223.255
72.44.32.0 - 72.44.63.255
75.101.128.0 - 75.101.255.255
79.125.0.0 - 79.125.127.255
87.238.80.0 - 87.238.87.255
103.4.8.0 - 103.4.15.255
107.20.0.0 - 107.23.255.255
122.248.192.0 - 122.248.255.255
174.129.0.0 - 174.129.255.255
175.41.128.0 - 175.41.255.255
176.32.64.0 - 176.32.127.255
176.34.0.0 - 176.34.255.255
177.71.128.0 - 177.71.255.255
178.236.0.0 - 178.236.15.255
184.72.0.0 - 184.73.255.255
184.169.128.0 - 184.169.255.255
199.255.192.0 - 199.255.195.255
204.236.128.0 - 204.236.255.255
205.251.192.0 - 205.251.255.255
207.171.160.0 - 207.171.191.255
216.182.224.0 - 216.182.239.255

lucy24




msg:4635103
 8:33 pm on Jan 4, 2014 (gmt 0)

54.192.0.0/10
54.222.0.0/15
54.224.0.0/11
54.240.0.0/12

b, c and d are contained within a. d is contained within c

/10 = 192-255
/15 = 222-223
/11 = 224-255
/12 = 240-255

You internalize it after a while. Honest, Don, you do.

keyplyr




msg:4635124
 10:11 pm on Jan 4, 2014 (gmt 0)


b, c and d are contained within a. d is contained within c


So you're saying 54.192.0.0/10 = 54.192.0.0 - 54.255.255.255?

I had thought 4.192.0.0/10 = 54.192.0.0 - 54.221.255.255

[edited by: keyplyr at 10:16 pm (utc) on Jan 4, 2014]

wilderness




msg:4635126
 10:22 pm on Jan 4, 2014 (gmt 0)

You internalize it after a while. Honest, Don, you do.


No need to internalize anything (besides, all my internalization space is occupied by widgets, and all that remains is sheer obstinence)

19[2-9]|2[0-5][0-9]

lucy24




msg:4635131
 11:11 pm on Jan 4, 2014 (gmt 0)

I had thought 54.192.0.0/10 = 54.192.0.0 - 54.221.255.255

Nothing is -221. Well, except 220-221, aka /15 /23 /31

/10 = /2 = /18 = /26 depending on which 0-255 block you're looking at

192 = 128+64 = binary 11000000
192/2 /10 /18 /26 = 11xxxxxx = 192-255
192/3 /11 /19 /27 = 110xxxxx = 192-223
192/4 /12 /20 /28 = 1100xxxx = 192-207
192/5 /13 /21 /29 = 11000xxx = 192-199

dstiles




msg:4635332
 8:54 pm on Jan 5, 2014 (gmt 0)

If you are running linux then look at GIP calculator.

For windows there are online and downloadable calculators from simple to "Why is it so complicated?"

A simple check, which I display for convenience on my IP-blocks manager:

Breakpoints within a /16 (ie from nnn.nnn.0.0 to nnn.nnn.255.255)...

/18: 0 64 128 192 256
/19: 0 32 64 96 128 160 192 224 256
/20: 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256
/21: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
(/22 is 4 /24s and /23 is 2 /24s)

Breakpoints within a /8 (ie from nnn.0.0.0 to nnn.255.255.255)...

/10: 0 64 128 192 256
/11: 0 32 64 96 128 160 192 224 256
/12: 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256
/13: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
(/14 is 4 /16s and /15 is 2 /16s)

keyplyr




msg:4635704
 7:33 am on Jan 7, 2014 (gmt 0)




Just a heads-up: Inside of AWS range 54.192.0.0/10 is Nokia Express, a mobile carrier bringing human traffic.

54.236.252.0/22
54.236.252.0 - 54.236.255.255

thetrasher




msg:4635750
 2:31 pm on Jan 7, 2014 (gmt 0)

Cloud-based Nokia Xpress Browser [en.wikipedia.org]

NetRange: 54.244.56.0 - 54.244.63.255
CIDR: 54.244.56.0/21
Name: AWS-XPRESSSERVICES1

NetRange: 54.236.252.0 - 54.236.255.255
CIDR: 54.236.252.0/22
Name: AWS-XPRESSSERVICES2

NetRange: 54.209.248.0 - 54.209.251.255
CIDR: 54.209.248.0/22
Name: AWS-XPRESSSERVICES3

This 44 message thread spans 2 pages: 44 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved