Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Using robot.txt to keep unwanted backlinking


ThomasB - 5:14 pm on May 21, 2004 (gmt 0)


I'm not quite sure what you're talking about but I'll try to answer what I guess you want to know. ;)

Having a robots.txt that prohibits real search engine spiders to go through your site mostly doesn't help to avoid backlinks. There are lots of bad bots out there masking as Mozilla or just not obeying the robots.txt. The only thing to handle this stuff is asking the site owners to remove the link. If they're stealing content, you can get the sites out by contacting the search engines.

I hope that met your question.


Thread source:: http://www.webmasterworld.com/robots_txt/389.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com