Over the last several months--maybe longer, because I’m slow on the uptake--I’ve been plagued by several patterns of robotic behavior.
IP: all over the map, mostly human broadband ranges
UA: fully plausible recent browser
Referer: either blank or something plausible like a search engine, no sign of referer spam
headers: fully humanoid, nothing missing, nothing fully suspicious
Pattern #1: Requests for the same page--no supporting files--10-20 or so clustered within a few seconds. All different IPs and UAs within each cluster. The frequency isn't high enough to make me suspect a DDoS attack from made-up IPs, especially when they never seem to come through as 429 (“too many requests”).
Pattern #2: Request for some random page, immediately followed by all supporting files from a different IP, most often 34.34.etcetera (blocked, of course).
Pattern #3: Request for one page and its supporting files (css and js) but not images.
Rarely, the IP turns out to be a server farm I hadn't previously known about (why, hello there, OVH! and you too Hetzner, didn’t know you lived there!) which can be happily blocked in perpetuity. But the great majority are human broadband IPs. Currently I deal with the latter by blocking the /24, because what else can I do. On any given day they are all different, but some are still in use when I do a three-month recheck. (In particular, when I look over each day’s clusters, a gratifying number are blocked, meaning that the IP has been here before.)
All these human IPs leave only two explanations. One, the IP itself might have nothing to do with the request--but, as noted above, I don't suspect DDoS activity. Two, there are a heck of a lot of people out there clicking on malware links.
Sigh.
