homepage Welcome to WebmasterWorld Guest from 54.167.185.110
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Suggestions On 'Low Value' Bots
Which bots provide little to no value to the web site.
netcent




msg:1527105
 5:43 pm on Jan 24, 2005 (gmt 0)

Hi Folks,

I was wondering if anyone had any suggestions (or a list) of bots that provide little to no value to the web site that it trolls. For example, the TurnItinBot. It will archive the content of your site, but only to allow teachers to compare student papers to see if kids are writing their own essays or copying from the web. A great service for teachers, but the indexing doesn't seem to do anything to promote the site it trolls.

Doug.

 

encyclo




msg:1527106
 6:24 pm on Jan 24, 2005 (gmt 0)

Welcome to WebmasterWorld [webmasterworld.com], Doug.

A good place to start is the robots.txt here at WebmasterWorld [webmasterworld.com]. However, it is important not to just blindly copy that list: you need to evaluate the usefullness of each bot in the context of your own site. Also, that list doesn't include bots which don't obey robots.txt, which you will need to ban by IP or user agent string.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved