homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Using robot.txt to keep unwanted backlinking
Is this an effective means
Daily Sparring

 4:57 pm on May 21, 2004 (gmt 0)

I have begun to hear recently that having backlinks from certain 'spam friendly' portals can be a very bad thing. Would using a robot.txt to keep these spiders out be effective, or would they just pull your data from your site anyways through other search engines?

I hope that makes sense, thanks in advance.



 5:14 pm on May 21, 2004 (gmt 0)

I'm not quite sure what you're talking about but I'll try to answer what I guess you want to know. ;)

Having a robots.txt that prohibits real search engine spiders to go through your site mostly doesn't help to avoid backlinks. There are lots of bad bots out there masking as Mozilla or just not obeying the robots.txt. The only thing to handle this stuff is asking the site owners to remove the link. If they're stealing content, you can get the sites out by contacting the search engines.

I hope that met your question.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved