GaryK - 2:54 pm on Dec 4, 2009 (gmt 0)

Because of my browser project I need to see how UAs behave so I usually let everything in and then only ban if the behavior is excessively egregious, like the Bing bots. It's because one of the things I do is recommend which UAs are ban-worthy. This version of Clam is now banned. But if a new one comes calling it'll be able to crawl until it does something bad enough to warrant my intervention.