Forum Moderators: open

Message Too Old, No Replies

List of 'BAD' bots?

Which one are they?

         

le_gber

9:53 am on Apr 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does anyone knows some bots that should be disallowed through the robots.txt file (ex. emails lookup and that kind of stuff?)

Thanks

Leo

Dreamquick

10:05 am on Apr 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



emails lookup and that kind of stuff

Most of the "bad" crawlers (such as email harvesters etc) don't respect robots.txt so adding their names into it wont have any effect since they will just ignore it anyway.

- Tony

pendanticist

8:57 pm on Apr 10, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



le_gber, do a Site Search [webmasterworld.com] using two or three variants like 'bots list', 'bad bots' and maybe 'bad bots list'. You'll find lots and lots of stuff.

For sure do a search for 'perfect .htaccess'.

Pendanticist.