Welcome to WebmasterWorld Guest from 54.196.244.206

Forum Moderators: goodroi

Message Too Old, No Replies

bad Bots

     
4:00 am on Oct 29, 2003 (gmt 0)

New User

10+ Year Member

joined:Oct 14, 2003
posts:12
votes: 0


Do bad bots follow robots.txt
what i mean is your server makes sure they are complying to your rules or the good faith of this email list bots?
4:17 am on Oct 29, 2003 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5800
votes: 64


cgj5204 - Welcome to Webmaster World! [webmasterworld.com]

Do bad bots follow robots.txt

Do criminals follow laws? Seriously, it is a bit of 'good faith' that bots follow the rules.

...your server makes sure they are complying to your rules[?]

Well, unless you are running your own server, no. When a robot shows they are not obeying the rules set in the robots.txt, or not even reading it, there are several courses of action we as webmasters can take, which is the topic of many a thread in the 'Webmaster General', 'Apache', 'Logging and Tracking' and 'Search Engine Identification' forums.
4:40 am on Oct 29, 2003 (gmt 0)

New User

10+ Year Member

joined:Oct 14, 2003
posts:12
votes: 0


Thank You
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members