GaryK
msg:1526731 | 12:31 am on Nov 4, 2002 (gmt 0) |
Check the site in my profile. One of the files I offer for download is a regularly updated robots.txt. For robots that don't obey robots.txt you can check the [Website Strippers] section of my browscap.ini file for the user agents I consider a nusiance. [edited by: GaryK at 12:36 am (utc) on Nov. 4, 2002]
|
Macguru
msg:1526732 | 12:32 am on Nov 4, 2002 (gmt 0) |
The scripting guys did a very nice job here. Almost perfect! ;) [webmasterworld.com...]
|
biggles
msg:1526733 | 1:33 am on Nov 4, 2002 (gmt 0) |
GaryK - thanks for the feedback, but no website listed in your profile. Macguru - wow, [webmasterworld.com ] what a thread. I'm not a code jockey and not confident about playing with the htaccess file - am I right thinking you use that for bots that don't respect robots.txt? Thanks
|
Macguru
msg:1526734 | 1:40 am on Nov 4, 2002 (gmt 0) |
Unfortunatly, yes. Bad bots dont care for the robots.txt file. My robots.txt files are quite basic. I just tell all good bots "*" places not to go. For the rest of creepy crawlers, an extra effort is required. I guess if you try .htacess on a test site, the scripting folks here will gladly help to set it up. The local web hosts I recommend to my clients, use it as added value. They run bullet proof servers and regularly update the list of banned bots and IPs.
|
biggles
msg:1526735 | 1:53 am on Nov 4, 2002 (gmt 0) |
GaryK - would you please send me a Sticky email with the URL. Thanks
|
GaryK
msg:1526736 | 1:55 am on Nov 4, 2002 (gmt 0) |
Done. :)
|
biggles
msg:1526737 | 1:56 am on Nov 4, 2002 (gmt 0) |
Macguru - thanks for the advice. Guess I'll have to be getting my head around htaccess, like it or not. :)
|
|
