Welcome to WebmasterWorld Guest from 50.19.190.144

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Amazon AWS Hosts Bad Bots

Continuation Thread

     
11:16 pm on May 16, 2013 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14621
votes: 85


This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.
9:57 pm on Jan 7, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


keyplr - it's cloud. The range could easily be some technical service or proxy service. If it's genuine mobile access then someone is being a bit naive.

And which thetrasher's link confirms. Nokia are obviously trying to do things on the cheap. If they provide a proper IP range they will get through.

It all comes down to trust. It's amazon: I don't.
2:32 am on Jan 8, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64




The mobile hits I see do contain "proxy" in the UA string.
12:57 am on Jan 9, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


Sample using a new Nokia, and you'll breathe-easy discovering why any "human" using one will soon deposit the Nokia in the dustbin.

As intuitive and effortlessly interactive as bats in the pantry.
10:01 am on Jan 9, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64




@dstiles
Well I poked a hole & let a few through. Look human to me, no issues yet. Keeping a close watch.
3:46 pm on Jan 30, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


Three hits today from a major new amazon range, registered in November.

54.72.0.0 - 54.95.255.255
4:59 pm on Jan 30, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64


Thanks dstiles, new for me.
8:06 pm on Jan 30, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12693
votes: 244


I had to look that up because it seemed so odd. Looks like the bottom half of 72-79 is still Merck-- at least this week-- but honestly, would anything bad happen if you just lock out the whole 54.0.0.0/8 and be done with it? I've never personally met anyone from <192.

:: idly wondering how Merck stock is doing these days ::
8:12 pm on Jan 30, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


The ranges either side of the amazon one are merck but I've left them alone, since they leave me alone. I have only two merck ranges listed and both are enabled.
2:56 pm on Feb 21, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255
CIDR: 54.176.0.0/12
OriginAS: AS16509
NetName: AMAZON-2011L
RegDate: 2013-11-25

That runs straight into one I logged Dec 2012...

54.192.0.0 - 54.255.255.255
6:38 pm on Feb 21, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64


Cool, that covers a lot
5:45 pm on Apr 5, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Aug 1, 2013
posts:1293
votes: 6


In looking at ranges inside 54.192.0.0/10, I notice a gap which appears to have been transferred to APNIC.

54.222.0.0 - 54.223.255.255 -- 54.222.0.0/16
netname: CHINANETCENTER
descr: Wangsu Science & Technology Co.,Ltd.

Thought it was worth mentioning.
7:39 pm on Apr 5, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


It has the same response: block it.

Incidentally, it's /15 not /16 :)
8:22 pm on Apr 5, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Aug 1, 2013
posts:1293
votes: 6


Oops, ran the query for /16 and forgot it returned /15. 54.192.0.0/10 works for me. ;) Thanks.
9:47 pm on Apr 5, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64



@webcentric - there's also Nokia Express mobile ISP in there:

54.209.248.0/22
54.209.248.0 - 54.209.251.255

54.236.252.0/22
54.236.252.0 - 54.236.255.255

54.244.56.0/21
54.244.56.0 - 54.244.63.255

However, that being said... I also block the entire /10
8:59 pm on Apr 27, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12693
votes: 244


Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255

I'd somehow overlooked this until I met one today :( Re-check in free lookup says that 96-175 is still Merck, leading to the question:

Has anyone, ever, met a legitimate human from anywhere in the 54 block? I don't mean in 1992 when this range was first allocated; I mean recently. Maybe if you've got a reputable medical-information site-- which I don't.

:: irresistible detour tells me that-- surprise! --Merck stock is performing respectably and even pays dividends ::
2:22 pm on July 3, 2014 (gmt 0)

New User

joined:July 3, 2014
posts: 1
votes: 0


New poster here. Found the site while trying to figure out an issue we had recently. Our company website got bombarded on the 1st of July for roughly 25 minutes - all IPs from ranges belonging to amazonaws. It looked like whatever or whoever it was scraped our site for all our product images. Burned a lot of our bandwidth too.

Ranges:

23.20.131.219 - 23.22.145.240

54.83.74.76 - 54.242.89.161

and one sole source at 107.20.19.114
2:51 pm on July 3, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64


Hi kazzo, welcome to WebmasterWorld.

These Amazon ranges have been listed earlier in this thread. You can use the site search utility at the top to find them, possibly searching for the A or A & B subnets.

23.20.131.219 - 23.22.145.240 is part of a greater Amazon range:
23.20.0.0 - 23.23.255.255
23.20.0.0/14

54.83.74.76 - 54.242.89.161 is part of a greater Amazon range:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

107.20.19.114 is part of a greater Amazon range:
107.20.0.0 - 107.23.255.255
107.20.0.0/14
4:32 am on July 4, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


# AISearchBot AmznAWS
deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

Blocking this fat tranche does NOT affect our Amazon affiliate status.
5:37 am on July 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64



deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

?
11:22 am on July 4, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


54. Inhuman as noted in this thread et al.
174.129.0.0/16 is all Amazon noise.

So why the ?
7:07 pm on July 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64


Wouldn't want to mislead others into thinking 54 is exclusively AWS, it isn't. Just a quick example...

Nokia Express mobile carrier:
54.209.248.0 - 54.209.251.255
54.209.248.0/22
54.236.252.0 - 54.236.255.255
54.236.252.0/22
54.244.56.0 - 54.244.63.255
54.244.56.0/21

There are others. Don't know about you, but I appreciate a large mobile customer base.
8:02 pm on July 4, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


Those ranges are still amazon-owned, though, and at least the first one says "services" which is not necessarily the same as "mobile broadband". It could be those ranges are actually non-public - eg nokia in-house or a VPN network.

Just an observation. Either way, they are blocked here.
9:25 pm on July 4, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64


@ dstiles - Personally, I get upwards of 60% mobile traffic on one site and over 40% of sales overall from mobile device users. Obviously Nokia popularity is geo specific so your particular user base may not be affected by blocking them, but this is a public forum often used as a knowledge base so IMO the correction to "54. Inhuman" was warranted.

Just a FYI - I block most all of 54, I just poke a few holes. Additionally, there are several other "holes" in 54. & 174.129. depending on your users.
11:07 pm on July 5, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2557
votes: 48


Just a note re:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

It needs 54.72.0.0/13 also to cover all of it, had to look it up today.
12:34 am on July 6, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64


Thanks not2easy

I was just pointing to where the earlier sub-range belonged.
11:05 pm on Aug 15, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


ec2-54-164-73-8.compute-1.amazonaws.com (a.k.a 54.164.73.8)
Manticore 0.3.1

robots.txt? NO

(Will also post as a standalone thread for UA-related comments.)
1:20 am on Aug 16, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5795
votes: 64




Thanks Pfui, didn't have this one:

54.160.0.0/12
54.160.0.0 - 54.175.255.255

***

So basically we have this:
54.160.0.0/12
54.160.0.0 - 54.175.255.255
54.176.0.0/12
4.176.0.0 - 54.191.255.255
54.192.0.0/10
54.192.0.0 - 54.255.255.255


Which can be efficiently minified to:
54.160.0.0 - 54.255.255.255
54.160.0.0/11
54.192.0.0/10
11:09 pm on Aug 19, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


FWIW: 54.167 has been busy of late, (ditto 54.166), including the following which doesn't really need a separate thread. Changing mixed-case to all-lower, it's clueless in its actions, and naming:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

(See also imminent thread: HubSpot Webcrawler)
1:27 am on Aug 20, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2557
votes: 48


I saw a string of visits today from their 54.164. range, strange behavior of one visit, go home, change shoes and come right back with slightly different IP numbers for 1 more .html, rinse, repeat.
UA: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.75 Safari/537.1"
4:08 pm on Nov 19, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3089
votes: 2


New amazon IP range registered in October:

NetRange: 54.144.0.0 - 54.159.255.255
CIDR: 54.144.0.0/12
Organization: Amazon Technologies Inc. (AT-88-Z)
OrgTechName: Amazon EC2 Network Operations

The latter suggests cloud.
This 108 message thread spans 4 pages: 108
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members