tangor - 3:16 am on Aug 28, 2010 (gmt 0)
I whitelist the big 4 (now three) which are of my interest and disallow all others. While there are many bad bots that ignore robots.txt, a very surprising number DO honor it. Create that invitation only robots.txt and you'll not have to wade through tons of bots to find the ones that do evil, or in other words, just deal with the misbehaved and let the rest take a peek at robots.txt, take it, and go away. If nothing else, robots.txt does act as a filter regarding behaved bots and those that misbehave.