Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies


what is this?

8:59 pm on Feb 10, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 12, 2002
votes: 0

I was looking at a sites robots.txt and it had the following:
User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: WebmasterWorld Extractor
Disallow: /

Sorry if this has been asked here a million times, I looked through this forum and searched on Google, but didn't see an explanation. Since these are also both on the robots.tx for this site I'm guessing that despite their names, they are not associated with this site?

10:33 pm on Feb 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
votes: 0

I have seen the content of WebmasterWorld threads and posts (word-for-word) appear on other sites, presumably without permission. I would assume these bots are doing the extracting of the Forum content (illegally).

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members