biggles - 12:21 am on Nov 4, 2002 (gmt 0)

Have been playing with the WebMaster World spider.txt checker and out of interest ran the webmasterworld.com/spider.txt through it. I was surprised at the number of excluded agents. Many seem to be email harvesters and site downloaders, which clearly makes sense.

Do people have a list of "nuisance" agents they suggest should be excluded by default for most sites?

Thanks