homepage Welcome to WebmasterWorld Guest from 54.166.113.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
bad Bots
cgj5204

10+ Year Member



 
Msg#: 179 posted 4:00 am on Oct 29, 2003 (gmt 0)

Do bad bots follow robots.txt
what i mean is your server makes sure they are complying to your rules or the good faith of this email list bots?

 

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 179 posted 4:17 am on Oct 29, 2003 (gmt 0)

cgj5204 - Welcome to Webmaster World! [webmasterworld.com]

Do bad bots follow robots.txt

Do criminals follow laws? Seriously, it is a bit of 'good faith' that bots follow the rules.

...your server makes sure they are complying to your rules[?]

Well, unless you are running your own server, no. When a robot shows they are not obeying the rules set in the robots.txt, or not even reading it, there are several courses of action we as webmasters can take, which is the topic of many a thread in the 'Webmaster General', 'Apache', 'Logging and Tracking' and 'Search Engine Identification' forums.

cgj5204

10+ Year Member



 
Msg#: 179 posted 4:40 am on Oct 29, 2003 (gmt 0)

Thank You

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved