homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Global "no follow" and discretionary "follow"

5+ Year Member

Msg#: 3826358 posted 7:23 pm on Jan 14, 2009 (gmt 0)

Hi, I just found out that Wikipedia implements "no follow" because of the existence of deadlinks (red links) due to internal hyperlinking.

As my website is also using the Mediawiki software like Wikipedia and since it is a one-year old site with a lot of deadlinks (red links), I wonder whether I can have a global "no follow" in the robot.txt file, followed by the following:

<meta name="Robots" content="index,follow">

on each article page, as soon as I manage to clear all the red links for each page.



WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3826358 posted 2:34 pm on Jan 15, 2009 (gmt 0)

I do not know about mediawiki but I do know that robots.txt can not apply nofollow to links. Robots.txt is a voluntary protocol that helps webmasters tell robots which files and folders should be accessed and which should not be accessed. If you block a file using robots.txt the search engine bots will not access it so they will not see the individual robots meta tag.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved