Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
Do people have a list of "nuisance" agents they suggest should be excluded by default for most sites?
[edited by: GaryK at 12:36 am (utc) on Nov. 4, 2002]
Macguru - wow, [webmasterworld.com ] what a thread. I'm not a code jockey and not confident about playing with the htaccess file - am I right thinking you use that for bots that don't respect robots.txt?
My robots.txt files are quite basic. I just tell all good bots "*" places not to go. For the rest of creepy crawlers, an extra effort is required.
I guess if you try .htacess on a test site, the scripting folks here will gladly help to set it up.
The local web hosts I recommend to my clients, use it as added value. They run bullet proof servers and regularly update the list of banned bots and IPs.