list of robots to disallow that actually obey robots.txt
koen
3:32 pm on Nov 28, 2005 (gmt 0)
It seems like the robots you wouldn't want are the ones who don't care about the robots.txt. So have people come up with a list of robots that actually are harmfull and do obey it?
