---- In the era of increasing numbers of bad bots is robots.txt irrelevant?
Kirby - 4:52 pm on Feb 20, 2006 (gmt 0)
I can count on one hand...
Don't have a bright idea at the moment to solve the weaknesses of robots.txt other that just whitelisting the allowed bots and everything else is blocked by default and enforced using .htaccess
This pretty much would solve the problem for me as I only care about those sending me traffic that dont do harm. I am willing to lose whatever miniscule amount of traffic that would be considered collateral damage.