homepage Welcome to WebmasterWorld Guest from 54.237.54.83
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 59 message thread spans 2 pages: < < 59 ( 1 [2]     
Amazon AWS Hosts Bad Bots
Continuation Thread
incrediBILL




msg:4574829
 11:16 pm on May 16, 2013 (gmt 0)

This is a continuation from the previous thread:
[webmasterworld.com...]

Post about spiders coming from Amazon's AWS hosting.

 

dstiles




msg:4635830
 9:57 pm on Jan 7, 2014 (gmt 0)

keyplr - it's cloud. The range could easily be some technical service or proxy service. If it's genuine mobile access then someone is being a bit naive.

And which thetrasher's link confirms. Nokia are obviously trying to do things on the cheap. If they provide a proper IP range they will get through.

It all comes down to trust. It's amazon: I don't.

keyplyr




msg:4635868
 2:32 am on Jan 8, 2014 (gmt 0)



The mobile hits I see do contain "proxy" in the UA string.

Angonasec




msg:4636058
 12:57 am on Jan 9, 2014 (gmt 0)

Sample using a new Nokia, and you'll breathe-easy discovering why any "human" using one will soon deposit the Nokia in the dustbin.

As intuitive and effortlessly interactive as bats in the pantry.

keyplyr




msg:4636137
 10:01 am on Jan 9, 2014 (gmt 0)



@dstiles
Well I poked a hole & let a few through. Look human to me, no issues yet. Keeping a close watch.

dstiles




msg:4641250
 3:46 pm on Jan 30, 2014 (gmt 0)

Three hits today from a major new amazon range, registered in November.

54.72.0.0 - 54.95.255.255

keyplyr




msg:4641278
 4:59 pm on Jan 30, 2014 (gmt 0)

Thanks dstiles, new for me.

lucy24




msg:4641308
 8:06 pm on Jan 30, 2014 (gmt 0)

I had to look that up because it seemed so odd. Looks like the bottom half of 72-79 is still Merck-- at least this week-- but honestly, would anything bad happen if you just lock out the whole 54.0.0.0/8 and be done with it? I've never personally met anyone from <192.

:: idly wondering how Merck stock is doing these days ::

dstiles




msg:4641312
 8:12 pm on Jan 30, 2014 (gmt 0)

The ranges either side of the amazon one are merck but I've left them alone, since they leave me alone. I have only two merck ranges listed and both are enabled.

dstiles




msg:4647482
 2:56 pm on Feb 21, 2014 (gmt 0)

Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255
CIDR: 54.176.0.0/12
OriginAS: AS16509
NetName: AMAZON-2011L
RegDate: 2013-11-25

That runs straight into one I logged Dec 2012...

54.192.0.0 - 54.255.255.255

keyplyr




msg:4647566
 6:38 pm on Feb 21, 2014 (gmt 0)

Cool, that covers a lot

webcentric




msg:4660734
 5:45 pm on Apr 5, 2014 (gmt 0)

In looking at ranges inside 54.192.0.0/10, I notice a gap which appears to have been transferred to APNIC.

54.222.0.0 - 54.223.255.255 -- 54.222.0.0/16
netname: CHINANETCENTER
descr: Wangsu Science & Technology Co.,Ltd.

Thought it was worth mentioning.

dstiles




msg:4660755
 7:39 pm on Apr 5, 2014 (gmt 0)

It has the same response: block it.

Incidentally, it's /15 not /16 :)

webcentric




msg:4660758
 8:22 pm on Apr 5, 2014 (gmt 0)

Oops, ran the query for /16 and forgot it returned /15. 54.192.0.0/10 works for me. ;) Thanks.

keyplyr




msg:4660764
 9:47 pm on Apr 5, 2014 (gmt 0)


@webcentric - there's also Nokia Express mobile ISP in there:

54.209.248.0/22
54.209.248.0 - 54.209.251.255

54.236.252.0/22
54.236.252.0 - 54.236.255.255

54.244.56.0/21
54.244.56.0 - 54.244.63.255

However, that being said... I also block the entire /10

lucy24




msg:4666332
 8:59 pm on Apr 27, 2014 (gmt 0)

Another large amazon range...

NetRange: 54.176.0.0 - 54.191.255.255

I'd somehow overlooked this until I met one today :( Re-check in free lookup says that 96-175 is still Merck, leading to the question:

Has anyone, ever, met a legitimate human from anywhere in the 54 block? I don't mean in 1992 when this range was first allocated; I mean recently. Maybe if you've got a reputable medical-information site-- which I don't.

:: irresistible detour tells me that-- surprise! --Merck stock is performing respectably and even pays dividends ::

kazzo




msg:4684847
 2:22 pm on Jul 3, 2014 (gmt 0)

New poster here. Found the site while trying to figure out an issue we had recently. Our company website got bombarded on the 1st of July for roughly 25 minutes - all IPs from ranges belonging to amazonaws. It looked like whatever or whoever it was scraped our site for all our product images. Burned a lot of our bandwidth too.

Ranges:

23.20.131.219 - 23.22.145.240

54.83.74.76 - 54.242.89.161

and one sole source at 107.20.19.114

keyplyr




msg:4684860
 2:51 pm on Jul 3, 2014 (gmt 0)

Hi kazzo, welcome to WebmasterWorld.

These Amazon ranges have been listed earlier in this thread. You can use the site search utility at the top to find them, possibly searching for the A or A & B subnets.

23.20.131.219 - 23.22.145.240 is part of a greater Amazon range:
23.20.0.0 - 23.23.255.255
23.20.0.0/14

54.83.74.76 - 54.242.89.161 is part of a greater Amazon range:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

107.20.19.114 is part of a greater Amazon range:
107.20.0.0 - 107.23.255.255
107.20.0.0/14

Angonasec




msg:4685022
 4:32 am on Jul 4, 2014 (gmt 0)

# AISearchBot AmznAWS
deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

Blocking this fat tranche does NOT affect our Amazon affiliate status.

keyplyr




msg:4685035
 5:37 am on Jul 4, 2014 (gmt 0)


deny from 23.20.0.0/14 50.16.0.0/14 50.112.0.0/16 54. 65.19.128.0/18 67.202.0.0/18 72.44.32.0/19 75.101.128.0/17 107.20.0.0/14 174.129. 184.72.0.0/15 184.169.128.0/17 204.236.128.0/17

?

Angonasec




msg:4685097
 11:22 am on Jul 4, 2014 (gmt 0)

54. Inhuman as noted in this thread et al.
174.129.0.0/16 is all Amazon noise.

So why the ?

keyplyr




msg:4685184
 7:07 pm on Jul 4, 2014 (gmt 0)

Wouldn't want to mislead others into thinking 54 is exclusively AWS, it isn't. Just a quick example...

Nokia Express mobile carrier:
54.209.248.0 - 54.209.251.255
54.209.248.0/22
54.236.252.0 - 54.236.255.255
54.236.252.0/22
54.244.56.0 - 54.244.63.255
54.244.56.0/21

There are others. Don't know about you, but I appreciate a large mobile customer base.

dstiles




msg:4685190
 8:02 pm on Jul 4, 2014 (gmt 0)

Those ranges are still amazon-owned, though, and at least the first one says "services" which is not necessarily the same as "mobile broadband". It could be those ranges are actually non-public - eg nokia in-house or a VPN network.

Just an observation. Either way, they are blocked here.

keyplyr




msg:4685216
 9:25 pm on Jul 4, 2014 (gmt 0)

@ dstiles - Personally, I get upwards of 60% mobile traffic on one site and over 40% of sales overall from mobile device users. Obviously Nokia popularity is geo specific so your particular user base may not be affected by blocking them, but this is a public forum often used as a knowledge base so IMO the correction to "54. Inhuman" was warranted.

Just a FYI - I block most all of 54, I just poke a few holes. Additionally, there are several other "holes" in 54. & 174.129. depending on your users.

not2easy




msg:4685356
 11:07 pm on Jul 5, 2014 (gmt 0)

Just a note re:
54.72.0.0 - 54.95.255.255
54.80.0.0/12

It needs 54.72.0.0/13 also to cover all of it, had to look it up today.

keyplyr




msg:4685362
 12:34 am on Jul 6, 2014 (gmt 0)

Thanks not2easy

I was just pointing to where the earlier sub-range belonged.

Pfui




msg:4695911
 11:05 pm on Aug 15, 2014 (gmt 0)

ec2-54-164-73-8.compute-1.amazonaws.com (a.k.a 54.164.73.8)
Manticore 0.3.1

robots.txt? NO

(Will also post as a standalone thread for UA-related comments.)

keyplyr




msg:4695917
 1:20 am on Aug 16, 2014 (gmt 0)



Thanks Pfui, didn't have this one:

54.160.0.0/12
54.160.0.0 - 54.175.255.255

***

So basically we have this:
54.160.0.0/12
54.160.0.0 - 54.175.255.255
54.176.0.0/12
4.176.0.0 - 54.191.255.255
54.192.0.0/10
54.192.0.0 - 54.255.255.255


Which can be efficiently minified to:
54.160.0.0 - 54.255.255.255
54.160.0.0/11
54.192.0.0/10

Pfui




msg:4696664
 11:09 pm on Aug 19, 2014 (gmt 0)

FWIW: 54.167 has been busy of late, (ditto 54.166), including the following which doesn't really need a separate thread. Changing mixed-case to all-lower, it's clueless in its actions, and naming:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31

(See also imminent thread: HubSpot Webcrawler)

not2easy




msg:4696682
 1:27 am on Aug 20, 2014 (gmt 0)

I saw a string of visits today from their 54.164. range, strange behavior of one visit, go home, change shoes and come right back with slightly different IP numbers for 1 more .html, rinse, repeat.
UA: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.75 Safari/537.1"

This 59 message thread spans 2 pages: < < 59 ( 1 [2]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved