incrediBILL - 8:00 pm on Feb 12, 2012 (gmt 0)
Why would you want to keep bots out? What happens if there is a new bot and you don't know about it? You may be losing customers?
Why keep bots out?
Ever spent a day or week chasing down copyright violators and wasting time and money hunting down site owners and sending DMCA requests?
Boring, costly, annoying, aggravating, expensive...
Depends on the bots and what they do.
I let a bunch of bots in, but if they don't identify themselves and display a legit reason to crawl, they don't get in.
How do you find new bots abusing your site?
You find new bots that could be beneficial the same way.
Personally, I have a script and get notified by my site when something new hits that has never been there before so I can see if it's worth letting it access my site.
The problem isn't which method is used, blacklisting or whitelisting, as both methods but both ways, issues either way. The problem is a lack of good tools to make either easy for the site owner to manage. That landscape is starting to change with better tools already available, they just aren't free.