Amazon AWS Hosts Bad Bots

Continuation Thread

[incrediBILL]

This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.

[dstiles]

keyplr - it's cloud. The range could easily be some technical service or proxy service. If it's genuine mobile access then someone is being a bit naive.

And which thetrasher's link confirms. Nokia are obviously trying to do things on the cheap. If they provide a proper IP range they will get through.

It all comes down to trust. It's amazon: I don't.

[keyplyr]

The mobile hits I see do contain "proxy" in the UA string.

[Angonasec]

Sample using a new Nokia, and you'll breathe-easy discovering why any "human" using one will soon deposit the Nokia in the dustbin.

As intuitive and effortlessly interactive as bats in the pantry.

[keyplyr]

@dstiles
Well I poked a hole & let a few through. Look human to me, no issues yet. Keeping a close watch.

[dstiles]

Three hits today from a major new amazon range, registered in November.

54.72.0.0 - 54.95.255.255

[keyplyr]

Thanks dstiles, new for me.

[lucy24]

I had to look that up because it seemed so odd. Looks like the bottom half of 72-79 is still Merck-- at least this week-- but honestly, would anything bad happen if you just lock out the whole 54.0.0.0/8 and be done with it? I've never personally met anyone from <192.

:: idly wondering how Merck stock is doing these days ::

[dstiles]

The ranges either side of the amazon one are merck but I've left them alone, since they leave me alone. I have only two merck ranges listed and both are enabled.

[dstiles]

Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255
CIDR: 54.176.0.0/12
OriginAS: AS16509
NetName: AMAZON-2011L
RegDate: 2013-11-25

That runs straight into one I logged Dec 2012...

54.192.0.0 - 54.255.255.255

[keyplyr]

Cool, that covers a lot

[webcentric]

In looking at ranges inside 54.192.0.0/10, I notice a gap which appears to have been transferred to APNIC.

54.222.0.0 - 54.223.255.255 -- 54.222.0.0/16
netname: CHINANETCENTER
descr: Wangsu Science & Technology Co.,Ltd.

Thought it was worth mentioning.

[dstiles]

It has the same response: block it.

Incidentally, it's /15 not /16 :)

[webcentric]

Oops, ran the query for /16 and forgot it returned /15. 54.192.0.0/10 works for me. ;) Thanks.

[keyplyr]

@webcentric - there's also Nokia Express mobile ISP in there:

54.209.248.0/22
54.209.248.0 - 54.209.251.255

54.236.252.0/22
54.236.252.0 - 54.236.255.255

54.244.56.0/21
54.244.56.0 - 54.244.63.255

However, that being said... I also block the entire /10

[lucy24]

Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255

I'd somehow overlooked this until I met one today :( Re-check in free lookup says that 96-175 is still Merck, leading to the question:

Has anyone, ever, met a legitimate human from anywhere in the 54 block? I don't mean in 1992 when this range was first allocated; I mean recently. Maybe if you've got a reputable medical-information site-- which I don't.

:: irresistible detour tells me that-- surprise! --Merck stock is performing respectably and even pays dividends ::

[kazzo]

New poster here. Found the site while trying to figure out an issue we had recently. Our company website got bombarded on the 1st of July for roughly 25 minutes - all IPs from ranges belonging to amazonaws. It looked like whatever or whoever it was scraped our site for all our product images. Burned a lot of our bandwidth too.

Ranges:

23.20.131.219 - 23.22.145.240

54.83.74.76 - 54.242.89.161

and one sole source at 107.20.19.114

[keyplyr]

Hi kazzo, welcome to WebmasterWorld.

These Amazon ranges have been listed earlier in this thread. You can use the site search utility at the top to find them, possibly searching for the A or A & B subnets.

23.20.131.219 - 23.22.145.240 is part of a greater Amazon range:
23.20.0.0 - 23.23.255.255
23.20.0.0/14

54.83.74.76 - 54.242.89.161 is part of a greater Amazon range:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

107.20.19.114 is part of a greater Amazon range:
107.20.0.0 - 107.23.255.255
107.20.0.0/14

[Angonasec]

# AISearchBot AmznAWS
deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

Blocking this fat tranche does NOT affect our Amazon affiliate status.

[keyplyr]

deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

?

[Angonasec]

54. Inhuman as noted in this thread et al.
174.129.0.0/16 is all Amazon noise.

So why the ?

[keyplyr]

Wouldn't want to mislead others into thinking 54 is exclusively AWS, it isn't. Just a quick example...

Nokia Express mobile carrier:
54.209.248.0 - 54.209.251.255
54.209.248.0/22
54.236.252.0 - 54.236.255.255
54.236.252.0/22
54.244.56.0 - 54.244.63.255
54.244.56.0/21

There are others. Don't know about you, but I appreciate a large mobile customer base.

[dstiles]

Those ranges are still amazon-owned, though, and at least the first one says "services" which is not necessarily the same as "mobile broadband". It could be those ranges are actually non-public - eg nokia in-house or a VPN network.

Just an observation. Either way, they are blocked here.

[keyplyr]

@ dstiles - Personally, I get upwards of 60% mobile traffic on one site and over 40% of sales overall from mobile device users. Obviously Nokia popularity is geo specific so your particular user base may not be affected by blocking them, but this is a public forum often used as a knowledge base so IMO the correction to "54. Inhuman" was warranted.

Just a FYI - I block most all of 54, I just poke a few holes. Additionally, there are several other "holes" in 54. & 174.129. depending on your users.

[not2easy]

Just a note re:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

It needs 54.72.0.0/13 also to cover all of it, had to look it up today.

[keyplyr]

Thanks not2easy

I was just pointing to where the earlier sub-range belonged.

[Pfui]

ec2-54-164-73-8.compute-1.amazonaws.com (a.k.a 54.164.73.8)
Manticore 0.3.1

robots.txt? NO

(Will also post as a standalone thread for UA-related comments.)

[keyplyr]

Thanks Pfui, didn't have this one:

54.160.0.0/12
54.160.0.0 - 54.175.255.255

***

So basically we have this:
54.160.0.0/12
54.160.0.0 - 54.175.255.255
54.176.0.0/12
4.176.0.0 - 54.191.255.255
54.192.0.0/10
54.192.0.0 - 54.255.255.255


Which can be efficiently minified to:
54.160.0.0 - 54.255.255.255
54.160.0.0/11
54.192.0.0/10

[Pfui]

FWIW: 54.167 has been busy of late, (ditto 54.166), including the following which doesn't really need a separate thread. Changing mixed-case to all-lower, it's clueless in its actions, and naming:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

(See also imminent thread: HubSpot Webcrawler)

[not2easy]

I saw a string of visits today from their 54.164. range, strange behavior of one visit, go home, change shoes and come right back with slightly different IP numbers for 1 more .html, rinse, repeat.
UA: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.75 Safari/537.1"

[dstiles]

New amazon IP range registered in October:

NetRange: 54.144.0.0 - 54.159.255.255
CIDR: 54.144.0.0/12
Organization: Amazon Technologies Inc. (AT-88-Z)
OrgTechName: Amazon EC2 Network Operations

The latter suggests cloud.