incrediBILL - 1:29 am on Apr 19, 2010 (gmt 0)
This has almost got me convinced its worth adopting IcrediBill's whitelist only approach to crawlers - the thing holding me back now is not "is it a good idea" , but "is it worth the effort"
Actually it's less effort.
Blacklisting means you analyze and chase everything and block new things you find.
Whitelisting means you only let in new things as needed, you never chase anything causing trouble because they're already blocked.