homepage Welcome to WebmasterWorld Guest from 54.145.209.80
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 62 message thread spans 3 pages: < < 62 ( 1 [2] 3 > >     
Amazon AWS Hosts Bad Bots
Continuation Thread
incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 11:16 pm on May 16, 2013 (gmt 0)

This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 9:57 pm on Jan 7, 2014 (gmt 0)

keyplr - it's cloud. The range could easily be some technical service or proxy service. If it's genuine mobile access then someone is being a bit naive.

And which thetrasher's link confirms. Nokia are obviously trying to do things on the cheap. If they provide a proper IP range they will get through.

It all comes down to trust. It's amazon: I don't.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 2:32 am on Jan 8, 2014 (gmt 0)



The mobile hits I see do contain "proxy" in the UA string.

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4574827 posted 12:57 am on Jan 9, 2014 (gmt 0)

Sample using a new Nokia, and you'll breathe-easy discovering why any "human" using one will soon deposit the Nokia in the dustbin.

As intuitive and effortlessly interactive as bats in the pantry.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 10:01 am on Jan 9, 2014 (gmt 0)



@dstiles
Well I poked a hole & let a few through. Look human to me, no issues yet. Keeping a close watch.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 3:46 pm on Jan 30, 2014 (gmt 0)

Three hits today from a major new amazon range, registered in November.

54.72.0.0 - 54.95.255.255

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 4:59 pm on Jan 30, 2014 (gmt 0)

Thanks dstiles, new for me.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4574827 posted 8:06 pm on Jan 30, 2014 (gmt 0)

I had to look that up because it seemed so odd. Looks like the bottom half of 72-79 is still Merck-- at least this week-- but honestly, would anything bad happen if you just lock out the whole 54.0.0.0/8 and be done with it? I've never personally met anyone from <192.

:: idly wondering how Merck stock is doing these days ::

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 8:12 pm on Jan 30, 2014 (gmt 0)

The ranges either side of the amazon one are merck but I've left them alone, since they leave me alone. I have only two merck ranges listed and both are enabled.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 2:56 pm on Feb 21, 2014 (gmt 0)

Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255
CIDR: 54.176.0.0/12
OriginAS: AS16509
NetName: AMAZON-2011L
RegDate: 2013-11-25

That runs straight into one I logged Dec 2012...

54.192.0.0 - 54.255.255.255

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 6:38 pm on Feb 21, 2014 (gmt 0)

Cool, that covers a lot

webcentric

WebmasterWorld Senior Member Top Contributors Of The Month



 
Msg#: 4574827 posted 5:45 pm on Apr 5, 2014 (gmt 0)

In looking at ranges inside 54.192.0.0/10, I notice a gap which appears to have been transferred to APNIC.

54.222.0.0 - 54.223.255.255 -- 54.222.0.0/16
netname: CHINANETCENTER
descr: Wangsu Science & Technology Co.,Ltd.

Thought it was worth mentioning.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 7:39 pm on Apr 5, 2014 (gmt 0)

It has the same response: block it.

Incidentally, it's /15 not /16 :)

webcentric

WebmasterWorld Senior Member Top Contributors Of The Month



 
Msg#: 4574827 posted 8:22 pm on Apr 5, 2014 (gmt 0)

Oops, ran the query for /16 and forgot it returned /15. 54.192.0.0/10 works for me. ;) Thanks.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 9:47 pm on Apr 5, 2014 (gmt 0)


@webcentric - there's also Nokia Express mobile ISP in there:

54.209.248.0/22
54.209.248.0 - 54.209.251.255

54.236.252.0/22
54.236.252.0 - 54.236.255.255

54.244.56.0/21
54.244.56.0 - 54.244.63.255

However, that being said... I also block the entire /10

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4574827 posted 8:59 pm on Apr 27, 2014 (gmt 0)

Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255

I'd somehow overlooked this until I met one today :( Re-check in free lookup says that 96-175 is still Merck, leading to the question:

Has anyone, ever, met a legitimate human from anywhere in the 54 block? I don't mean in 1992 when this range was first allocated; I mean recently. Maybe if you've got a reputable medical-information site-- which I don't.

:: irresistible detour tells me that-- surprise! --Merck stock is performing respectably and even pays dividends ::

kazzo



 
Msg#: 4574827 posted 2:22 pm on Jul 3, 2014 (gmt 0)

New poster here. Found the site while trying to figure out an issue we had recently. Our company website got bombarded on the 1st of July for roughly 25 minutes - all IPs from ranges belonging to amazonaws. It looked like whatever or whoever it was scraped our site for all our product images. Burned a lot of our bandwidth too.

Ranges:

23.20.131.219 - 23.22.145.240

54.83.74.76 - 54.242.89.161

and one sole source at 107.20.19.114

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 2:51 pm on Jul 3, 2014 (gmt 0)

Hi kazzo, welcome to WebmasterWorld.

These Amazon ranges have been listed earlier in this thread. You can use the site search utility at the top to find them, possibly searching for the A or A & B subnets.

23.20.131.219 - 23.22.145.240 is part of a greater Amazon range:
23.20.0.0 - 23.23.255.255
23.20.0.0/14

54.83.74.76 - 54.242.89.161 is part of a greater Amazon range:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

107.20.19.114 is part of a greater Amazon range:
107.20.0.0 - 107.23.255.255
107.20.0.0/14

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4574827 posted 4:32 am on Jul 4, 2014 (gmt 0)

# AISearchBot AmznAWS
deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

Blocking this fat tranche does NOT affect our Amazon affiliate status.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 5:37 am on Jul 4, 2014 (gmt 0)


deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

?

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4574827 posted 11:22 am on Jul 4, 2014 (gmt 0)

54. Inhuman as noted in this thread et al.
174.129.0.0/16 is all Amazon noise.

So why the ?

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 7:07 pm on Jul 4, 2014 (gmt 0)

Wouldn't want to mislead others into thinking 54 is exclusively AWS, it isn't. Just a quick example...

Nokia Express mobile carrier:
54.209.248.0 - 54.209.251.255
54.209.248.0/22
54.236.252.0 - 54.236.255.255
54.236.252.0/22
54.244.56.0 - 54.244.63.255
54.244.56.0/21

There are others. Don't know about you, but I appreciate a large mobile customer base.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 8:02 pm on Jul 4, 2014 (gmt 0)

Those ranges are still amazon-owned, though, and at least the first one says "services" which is not necessarily the same as "mobile broadband". It could be those ranges are actually non-public - eg nokia in-house or a VPN network.

Just an observation. Either way, they are blocked here.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 9:25 pm on Jul 4, 2014 (gmt 0)

@ dstiles - Personally, I get upwards of 60% mobile traffic on one site and over 40% of sales overall from mobile device users. Obviously Nokia popularity is geo specific so your particular user base may not be affected by blocking them, but this is a public forum often used as a knowledge base so IMO the correction to "54. Inhuman" was warranted.

Just a FYI - I block most all of 54, I just poke a few holes. Additionally, there are several other "holes" in 54. & 174.129. depending on your users.

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 11:07 pm on Jul 5, 2014 (gmt 0)

Just a note re:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

It needs 54.72.0.0/13 also to cover all of it, had to look it up today.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 12:34 am on Jul 6, 2014 (gmt 0)

Thanks not2easy

I was just pointing to where the earlier sub-range belonged.

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4574827 posted 11:05 pm on Aug 15, 2014 (gmt 0)

ec2-54-164-73-8.compute-1.amazonaws.com (a.k.a 54.164.73.8)
Manticore 0.3.1

robots.txt? NO

(Will also post as a standalone thread for UA-related comments.)

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 1:20 am on Aug 16, 2014 (gmt 0)



Thanks Pfui, didn't have this one:

54.160.0.0/12
54.160.0.0 - 54.175.255.255

***

So basically we have this:
54.160.0.0/12
54.160.0.0 - 54.175.255.255
54.176.0.0/12
4.176.0.0 - 54.191.255.255
54.192.0.0/10
54.192.0.0 - 54.255.255.255


Which can be efficiently minified to:
54.160.0.0 - 54.255.255.255
54.160.0.0/11
54.192.0.0/10

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4574827 posted 11:09 pm on Aug 19, 2014 (gmt 0)

FWIW: 54.167 has been busy of late, (ditto 54.166), including the following which doesn't really need a separate thread. Changing mixed-case to all-lower, it's clueless in its actions, and naming:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

(See also imminent thread: HubSpot Webcrawler)

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 1:27 am on Aug 20, 2014 (gmt 0)

I saw a string of visits today from their 54.164. range, strange behavior of one visit, go home, change shoes and come right back with slightly different IP numbers for 1 more .html, rinse, repeat.
UA: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.75 Safari/537.1"

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4574827 posted 4:08 pm on Nov 19, 2014 (gmt 0)

New amazon IP range registered in October:

NetRange: 54.144.0.0 - 54.159.255.255
CIDR: 54.144.0.0/12
Organization: Amazon Technologies Inc. (AT-88-Z)
OrgTechName: Amazon EC2 Network Operations

The latter suggests cloud.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4574827 posted 4:16 am on Nov 20, 2014 (gmt 0)

Just a heads-up: I am blocking the Amazon ranges listed in these forums. Occasionally I get reports from Kindle users they are being blocked or that they do not see my site's images, or that certain functions of my sites are not performing properly. These reports only from Kindle users.

The IPs (if they include them in the report) are spread across various Amazon ranges and I now suspect these are being assigned dynamically, for various reasons.

There always were tales of Amazon caching images and doing other resource saving measures for their Kindle clients, but until lately I didn't notice any backlash to blocking AWS. Either my sites are becoming more popular with Kindle users, or Kindle users are growing to effectively make this kind of impact.

This 62 message thread spans 3 pages: < < 62 ( 1 [2] 3 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved