| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Strange bot keep on visiting my site and don't obey my robots.txt file|
What to do?
| 7:40 pm on Apr 22, 2004 (gmt 0)|
The following bots keep on visiting my site dispite my rules in the robots.txt file. Here they are:
What do I do?
| 8:01 pm on Apr 22, 2004 (gmt 0)|
Ban bad bots in your .htaccess file. The major SE bots usually follow robots.txt, but linkwalkers, e-mail harvesters, test bots, etc. usually don't even check robots.txt.
Here's an extensive discussion:
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved