homepage Welcome to WebmasterWorld Guest from 54.227.20.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
WebmasterWorldForumBot?
what is this?
Trisha

10+ Year Member



 
Msg#: 280 posted 8:59 pm on Feb 10, 2004 (gmt 0)

I was looking at a sites robots.txt and it had the following:
User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: WebmasterWorld Extractor
Disallow: /

Sorry if this has been asked here a million times, I looked through this forum and searched on Google, but didn't see an explanation. Since these are also both on the robots.tx for this site I'm guessing that despite their names, they are not associated with this site?

 

pleeker

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 280 posted 10:33 pm on Feb 10, 2004 (gmt 0)

I have seen the content of WebmasterWorld threads and posts (word-for-word) appear on other sites, presumably without permission. I would assume these bots are doing the extracting of the Forum content (illegally).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved