-- Search Engine Spider and User Agent Identification
---- International Data Centers
incrediBILL - 6:40 am on Sep 16, 2013 (gmt 0)
Some things are pretty obvious, esp. for those that return reverse DNS data.
Guess my real problem is trying to make sense out of some of the APNIC records.
Some countries are pretty easy, others are a real PITA, especially if you don't know some of the local terms and such. Although I've been doing this for many years I still find it a real learning experience and often I find Asian countries the hardest as they don't deliver the data as granular as needed and/or using the terms required to make simple decisions.
I've resorted to blocking entire providers (like a Comcast equivalent) or worse case entire countries just out of frustration.
I take a few random IP scans using umit.
Do you do this via a proxy or directly from your server's IP?
Without using a proxy, I'd be worried about having my IP flagged as a potential security risk for scanning people's IPs.