homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 59 message thread spans 2 pages: < < 59 ( 1 [2]     
Amazon AWS Hosts Bad Bots
Continuation Thread

 11:16 pm on May 16, 2013 (gmt 0)

This is a continuation from the previous thread:

Post about spiders coming from Amazon's AWS hosting.



 9:57 pm on Jan 7, 2014 (gmt 0)

keyplr - it's cloud. The range could easily be some technical service or proxy service. If it's genuine mobile access then someone is being a bit naive.

And which thetrasher's link confirms. Nokia are obviously trying to do things on the cheap. If they provide a proper IP range they will get through.

It all comes down to trust. It's amazon: I don't.


 2:32 am on Jan 8, 2014 (gmt 0)

The mobile hits I see do contain "proxy" in the UA string.


 12:57 am on Jan 9, 2014 (gmt 0)

Sample using a new Nokia, and you'll breathe-easy discovering why any "human" using one will soon deposit the Nokia in the dustbin.

As intuitive and effortlessly interactive as bats in the pantry.


 10:01 am on Jan 9, 2014 (gmt 0)

Well I poked a hole & let a few through. Look human to me, no issues yet. Keeping a close watch.


 3:46 pm on Jan 30, 2014 (gmt 0)

Three hits today from a major new amazon range, registered in November. -


 4:59 pm on Jan 30, 2014 (gmt 0)

Thanks dstiles, new for me.


 8:06 pm on Jan 30, 2014 (gmt 0)

I had to look that up because it seemed so odd. Looks like the bottom half of 72-79 is still Merck-- at least this week-- but honestly, would anything bad happen if you just lock out the whole and be done with it? I've never personally met anyone from <192.

:: idly wondering how Merck stock is doing these days ::


 8:12 pm on Jan 30, 2014 (gmt 0)

The ranges either side of the amazon one are merck but I've left them alone, since they leave me alone. I have only two merck ranges listed and both are enabled.


 2:56 pm on Feb 21, 2014 (gmt 0)

Another large amazon range...

NetRange: -
OriginAS: AS16509
NetName: AMAZON-2011L
RegDate: 2013-11-25

That runs straight into one I logged Dec 2012... -


 6:38 pm on Feb 21, 2014 (gmt 0)

Cool, that covers a lot


 5:45 pm on Apr 5, 2014 (gmt 0)

In looking at ranges inside, I notice a gap which appears to have been transferred to APNIC. - --
descr: Wangsu Science & Technology Co.,Ltd.

Thought it was worth mentioning.


 7:39 pm on Apr 5, 2014 (gmt 0)

It has the same response: block it.

Incidentally, it's /15 not /16 :)


 8:22 pm on Apr 5, 2014 (gmt 0)

Oops, ran the query for /16 and forgot it returned /15. works for me. ;) Thanks.


 9:47 pm on Apr 5, 2014 (gmt 0)

@webcentric - there's also Nokia Express mobile ISP in there: - - -

However, that being said... I also block the entire /10


 8:59 pm on Apr 27, 2014 (gmt 0)

Another large amazon range...

NetRange: -

I'd somehow overlooked this until I met one today :( Re-check in free lookup says that 96-175 is still Merck, leading to the question:

Has anyone, ever, met a legitimate human from anywhere in the 54 block? I don't mean in 1992 when this range was first allocated; I mean recently. Maybe if you've got a reputable medical-information site-- which I don't.

:: irresistible detour tells me that-- surprise! --Merck stock is performing respectably and even pays dividends ::


 2:22 pm on Jul 3, 2014 (gmt 0)

New poster here. Found the site while trying to figure out an issue we had recently. Our company website got bombarded on the 1st of July for roughly 25 minutes - all IPs from ranges belonging to amazonaws. It looked like whatever or whoever it was scraped our site for all our product images. Burned a lot of our bandwidth too.

Ranges: - -

and one sole source at


 2:51 pm on Jul 3, 2014 (gmt 0)

Hi kazzo, welcome to WebmasterWorld.

These Amazon ranges have been listed earlier in this thread. You can use the site search utility at the top to find them, possibly searching for the A or A & B subnets. - is part of a greater Amazon range: - - is part of a greater Amazon range: - is part of a greater Amazon range: -


 4:32 am on Jul 4, 2014 (gmt 0)

# AISearchBot AmznAWS
deny from 54. 174.129.

Blocking this fat tranche does NOT affect our Amazon affiliate status.


 5:37 am on Jul 4, 2014 (gmt 0)

deny from 54. 174.129.



 11:22 am on Jul 4, 2014 (gmt 0)

54. Inhuman as noted in this thread et al. is all Amazon noise.

So why the ?


 7:07 pm on Jul 4, 2014 (gmt 0)

Wouldn't want to mislead others into thinking 54 is exclusively AWS, it isn't. Just a quick example...

Nokia Express mobile carrier: - - -

There are others. Don't know about you, but I appreciate a large mobile customer base.


 8:02 pm on Jul 4, 2014 (gmt 0)

Those ranges are still amazon-owned, though, and at least the first one says "services" which is not necessarily the same as "mobile broadband". It could be those ranges are actually non-public - eg nokia in-house or a VPN network.

Just an observation. Either way, they are blocked here.


 9:25 pm on Jul 4, 2014 (gmt 0)

@ dstiles - Personally, I get upwards of 60% mobile traffic on one site and over 40% of sales overall from mobile device users. Obviously Nokia popularity is geo specific so your particular user base may not be affected by blocking them, but this is a public forum often used as a knowledge base so IMO the correction to "54. Inhuman" was warranted.

Just a FYI - I block most all of 54, I just poke a few holes. Additionally, there are several other "holes" in 54. & 174.129. depending on your users.


 11:07 pm on Jul 5, 2014 (gmt 0)

Just a note re: -

It needs also to cover all of it, had to look it up today.


 12:34 am on Jul 6, 2014 (gmt 0)

Thanks not2easy

I was just pointing to where the earlier sub-range belonged.


 11:05 pm on Aug 15, 2014 (gmt 0)

ec2-54-164-73-8.compute-1.amazonaws.com (a.k.a
Manticore 0.3.1

robots.txt? NO

(Will also post as a standalone thread for UA-related comments.)


 1:20 am on Aug 16, 2014 (gmt 0)

Thanks Pfui, didn't have this one: -


So basically we have this: - - -

Which can be efficiently minified to: -


 11:09 pm on Aug 19, 2014 (gmt 0)

FWIW: 54.167 has been busy of late, (ditto 54.166), including the following which doesn't really need a separate thread. Changing mixed-case to all-lower, it's clueless in its actions, and naming:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

(See also imminent thread: HubSpot Webcrawler)


 1:27 am on Aug 20, 2014 (gmt 0)

I saw a string of visits today from their 54.164. range, strange behavior of one visit, go home, change shoes and come right back with slightly different IP numbers for 1 more .html, rinse, repeat.
UA: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.75 Safari/537.1"

This 59 message thread spans 2 pages: < < 59 ( 1 [2]
Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved