Brett_Tabke - 9:23 pm on Oct 28, 2003 (gmt 0)

Let's be honest: the robots.txt standard is useless at stopping rogue bots. I want to do a complete ban on all bots but the good search engine bots. How can you do that with thousands of bot names you don't know? Ban 'em all, and then let the good bots in via JD's script.

What your competitors don't know - can't hurt you, but if they get too snoopy, it can hurt them.