-- Search Engine Spider and User Agent Identification
---- Filtering Out Really Hard To Find Bad Bots
incrediBILL - 5:35 am on Jan 19, 2013 (gmt 0)
I think there are more "broadband" ISP ranges than server ranges.
That's why I'm doing it with reverse DNS results at the moment as that list is a lot smaller than the large lists small fragmented IP ranges.
Also, when you're using reverse DNS you can automatically compile IP range lists as you go along based on actual accesses and do a WHOIS sanity check later if something looks suspicious.
I would also point out that a LOT of "broadband" IPs I block are run by botnets:
Those IPs are handled on an individual basis because you couldn't block all of Comcast or Cox unless you wanted to lose a ton of customers. Likewise, compromised Android, which I've yet to see as I'm suspecting you're seeing a fake user agent string, if it's legit Android is typically using a 3G or 4G IP pool and you can't really block those except for a few hours at most unless you don't want mobile customers either.