motorhaven - 3:01 am on Mar 25, 2012 (gmt 0)
Looks like its simply not going to be easy. I'll go back to my dynamic robots.txt/honey pot project and work on that some more. It also has measures in place to detect what looks like a normal browser but isn't. I just wish I could do it more proactively with a decent whitelist rather than so reactively. I hate leeches!