Netcraft Recap

[Pfui]

Seeing more of this still-rude bot lately, two to three times/week on some sites, apparently hitting IPs in sequence. Alas, it's still:

- using multiple UAs;
- going for root or an Apache server (not web directory) file;
- log-spamming;
- only from AWS (in recent years), and, of course...
- not requesting robots.txt

ec2-107-22-65-54.compute-1.amazonaws.com
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [e-mail])
URI: /
robots.txt? NO

ec2-107-20-93-129.compute-1.amazonaws.com
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [e-mail])
URI: /icons/apache_pb.gif
robots.txt? NO

ec2-50-17-19-35.compute-1.amazonaws.com
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
URI: /
robots.txt? NO
Fake REF? Yes: http://www.netcraft.com/survey/

[keyplyr]

I don't always say thanks Pfui, but I do appreciate the time you spend accounting for this never ending mötley crüe coming from AWS. I double check them against my defenses and occasionally find a hole I didn't know I had, so gracias :)

[Staffa]

Ditto, so thanks also Pfui.
I have all known AWS ranges blocked but they can always come up with a new one :o)