Msg#: 4192676 posted 1:42 am on Aug 28, 2010 (gmt 0)
it depends partly on how far you want to go and how much information about your configuration and your definition of a "good bot" you want to expose. this thread should cover many of the issues and methods.
Msg#: 4192676 posted 3:16 am on Aug 28, 2010 (gmt 0)
I whitelist the big 4 (now three) which are of my interest and disallow all others. While there are many bad bots that ignore robots.txt, a very surprising number DO honor it. Create that invitation only robots.txt and you'll not have to wade through tons of bots to find the ones that do evil, or in other words, just deal with the misbehaved and let the rest take a peek at robots.txt, take it, and go away. If nothing else, robots.txt does act as a filter regarding behaved bots and those that misbehave.