-- Search Engine Spider and User Agent Identification
---- Stale bad bot lists
lucy24 - 2:45 am on Mar 25, 2012 (gmt 0)
Many others prefer white-listing over black-listing entirely.
Every time I go to take a closer look at whitelisting instructions, it turns out to involve a robot identifying itself upfront, for example by dutifully visiting some cranked-up version of robots.txt, or by not spoofing a human UA. So the easiest ones to trap are the stupid and/or honest robots.
Now, what would be nice-- and might even be sort-of possible-- is a current list of IP ranges showing where the servers live.
:: pause to contemplate mental picture of a stale robot ::