Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- New Thought to an Old Thread

phranque - 1:34 am on Oct 11, 2012 (gmt 0)


the process of "crawling the url database" is completely divorced from "assigning link equity".
robots.txt is about excluding crawlers.
excluding link equity would require a separate protocol.

Thread source:: http://www.webmasterworld.com/robots_txt/4506639.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com