blocking amazon networks

[markRg]

Hello
I am seeing thousands of requests from Amazon's network. Do you think it would be a good idea to block all Amazon networks on my site?

[not2easy]

Hi markRg and welcome to WebmasterWorld [webmasterworld.com]

There are several (probably dozens) of threads around here that list the CIDRs of various Amazon servers. If you're using Cloudflare, Amazon's ASN might be easier. Either way, the search in the header menu bar can help you find them.

[markRg]

Thank you for your reply.
Yes, it's not hard to block the Amazon network by ASN.
The problem I see is that about 0.1% of the traffic from Amazon has a referrer from Google

so it's good idea to block amazon ASN or not ?

[not2easy]

A lot of those Google 'referrers' are added by the bot runners. If you examine your logs you will see that no referer (sic) is logged, it is simply added to the request string in their script. Notice whether those claiming a Google referral actually request all the assets on the page (images, js, etc.). You could also check whether their IP is an ISP or a server.

[markRg]

I understand that this 0.1% of users looks like normal clients. They could be users who are using their Amazon servers as a VPN server or something like that. I'm not sure if I want to block them

[Brett_Tabke]

The hard part is tracking down on the IP's. Man they have a bunch.

I've heard the best way to do it, is use CloudFlare and turn it on hyper aggressive mode...

[thecoalman]

Duckduckgo is/was hosted on AWS, doing some quick research looks like they moved to MS Azure. Whatever the case the point is you need to be careful with ASN blocks. I know I blocked a network once where one of my users was utilizing VPN on same network unrelated to the IP's causing issue.

If you are using Cloudflare they use AND/OR so you can create some pretty specific rules. Not the exact syntax but something like:

ASN is equal to Amazon's ASN
AND
User agent does not equal Duckduckgo
OR........

This can also be entirely avoided by using skip rule before blocking rules.

[Sutin]

FYI: I get hit by Amazon IPs that have no corresponding ASN. I haven't yet encountered any other cloud company with non-ASN IP ranges.

[markRg]

could your provide any example ? interesting to see

[Sutin]

IPs with no ASNs are not rare. Digging through my logs is a PITA, but here is my .htaccess firewall for no-ASN AWS ranges (that I know of):

RewriteCond %{REMOTE_ADDR} ^18\.(3[2-9]|[4-9]\d|[12]\d\d)\.  [OR] # 18.32.0.0 - 18.255.255.255 no ASN!
RewriteCond %{REMOTE_ADDR} ^98\.6[8-9]\.   [OR] # 98.68.0.0 - 98.69.255.255 no ASN!
RewriteCond %{REMOTE_ADDR} ^110\.238\.(12[89]|1[3-9]\d|2\d\d)\. [OR] # 110.238.128.0 - 110.238.255.255 no ASN!
RewriteCond %{REMOTE_ADDR} ^119\.13\.(12[89]|13\d|14[0-3])\. [OR] # 119.13.128.0 - 119.13.143.255 no ASN!
RewriteCond %{REMOTE_ADDR} ^136\.[12]\.   [NC] # 136.1.0.0 - 136.2.255.255 no ASN!

[markRg]

Wow, someone is still using .htaccess, I’m surprised

[thecoalman]

What are you using to look them up? The tools I'm using are reporting 16509

[markRg]

>What are you using to look them up? The tools I'm using are reporting 16509
this is good place, but first read this: [bgp.tools...]

wget https://bgp.tools/table.jsonl ; grep ':16509' table.jsonl

[thecoalman]

Thanks for the link, bookmarked, but I was asking Sutin.

[Sutin]

whois.radb.net or whois.iana.org

[Jonesy]

I run this in a cron job each night to get a list of all Amazon IPs/CIDRs.
See <https://ip-ranges.amazonaws.com/ip-ranges.json>
It's also used to build (with other major lists) a "master" pf blocklist in my VPS.

#!/bin/sh
#
## Retrieve Amazon's IPv4 CIDR list.

## First light 25-Dec-23 ----- Marvin Jones
##
## Sort using -V to get the file in numerical CIDR seq.
## Sort using -u because Amazon's list is chock full
## of duplicate CIDRs.
## There are also overlapping CIDRs, but those will
## be cleaned up by `aggregate` when we build the
## MASTER_block_lst

# Get and process the Amazon's CIDRs
/usr/local/bin/curl -s https://ip-ranges.amazonaws.com/ip-ranges.json \
 | /usr/local/bin/jq -r '.prefixes | .[].ip_prefix' \
 | /usr/bin/grep -v null  \
 | /usr/bin/sort -uV > /home/REDACTED/pf_files/block_amazon

exit 0

[thecoalman]

Looks like they might of switched to MS cloud services but Duckduckgo was/is on AWS.

[mack]

AWS and recently Microsoft datacenters are a massive issue. I really think it is time the providers started deploying systems to prevent abuse. There are patterns of outgoing traffic that are never going to be normal. When this happens, throttle that action. It seems unfair that hosts and end users need to be on the defensive against server providers.

I ended up writing a script that runs on a cron every hour. It reads my log file for any IP that makes more than 50 requests. It stores these in an array and checks them against AbuseIPDB. If it scores more than 80% abuse it gets blocked by iptables.

Mack.

[Edge]

I’m late to this discussion whoever I submit that one should be careful blocking access to visitors from AWS.

I agree that there is unwanted traffic from the network however not all traffic is undesired. There are many companies and other entities that utilize the AWS network for organizational access to the internet. Also, some (many?) VPS user is funneled through AWS.

I have active bot traps that generally IP block scraper bots well as well as other methods to separate good from bad AWS traffic

[Kendo]

AWS and recently Microsoft datacenters are a massive issue.

You can add Google servers to that list - not all are Google bots.

Why would a web server be visiting our web pages?

[mack]

Kendo you are right. I am also seeing a significant increase from Google data centres. As for why servers would be visting your sites. Port sniffing, looking for WP sites to exploit. Servers with their fast connections are perfect for this, unfortunately.

Mack.

[Kendo]

I have been using [ipinfo.io...] to check their location and history, and I have been finding that most IPs that get my attention have been reported by other webmasters as abusive. With so many reports and complaints, one would think that those hosts would take action.

[Edge]

I thought of this thread today given that "Amazon Web Services" has disruptions...

Be careful who you block....

[lucy24]

Dang, I wish someone had taken the time to explain this:

Wow, someone is still using .htaccess, I’m surprised

If one is on shared hosting on an Apache server, what else would one use for access control?

fwiw: In the rare cases where an acceptable robot comes from an unsavory neighborhood, I use a “two steps forward, one back” approach:

SetEnvIf Remote_Addr ^{numbers here} bad_range=$0
BrowserMatch {name here} !bad_range

Require env bad_range