Does anyone knows some bots that should be disallowed through the robots.txt file (ex. emails lookup and that kind of stuff?)
Thanks
Leo
Dreamquick
10:05 am on Apr 9, 2003 (gmt 0)
emails lookup and that kind of stuff
Most of the "bad" crawlers (such as email harvesters etc) don't respect robots.txt so adding their names into it wont have any effect since they will just ignore it anyway.
- Tony
pendanticist
8:57 pm on Apr 10, 2003 (gmt 0)
le_gber, do a Site Search [webmasterworld.com] using two or three variants like 'bots list', 'bad bots' and maybe 'bad bots list'. You'll find lots and lots of stuff.