Welcome to WebmasterWorld Guest from 54.196.175.173

Forum Moderators: goodroi

Message Too Old, No Replies

bad Bots

     

cgj5204

4:00 am on Oct 29, 2003 (gmt 0)

10+ Year Member



Do bad bots follow robots.txt
what i mean is your server makes sure they are complying to your rules or the good faith of this email list bots?

keyplyr

4:17 am on Oct 29, 2003 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



cgj5204 - Welcome to Webmaster World! [webmasterworld.com]

Do bad bots follow robots.txt

Do criminals follow laws? Seriously, it is a bit of 'good faith' that bots follow the rules.

...your server makes sure they are complying to your rules[?]

Well, unless you are running your own server, no. When a robot shows they are not obeying the rules set in the robots.txt, or not even reading it, there are several courses of action we as webmasters can take, which is the topic of many a thread in the 'Webmaster General', 'Apache', 'Logging and Tracking' and 'Search Engine Identification' forums.

cgj5204

4:40 am on Oct 29, 2003 (gmt 0)

10+ Year Member



Thank You
 

Featured Threads

Hot Threads This Week

Hot Threads This Month