trillianjedi - 10:58 am on Nov 20, 2005 (gmt 0)

Seems wrong to me to try and use robots.txt to ban rogue crawlers - the truly rogue crawlers don't obey it anyway.

Does robots.txt not get parsed in sequence - i.e. you do the inverse of what you did previously - allow the crawlers you specifically want at the top of the file, and the last line in the file is ban everything?