Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

New Thought to an Old Thread

Before you reply, read the entire post including the thread link

9:51 pm on Oct 10, 2012 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2004
posts: 565
votes: 1

I would like to amend my original post [webmasterworld.com] to pose the following -

Rather than disavowing sites through the various SE’s webmaster tool sets, why not simply put the information in a single location; such as the robots.txt file? I know this will be like moving mountains, but read on.

If there was a robots.txt rule, which when a bot see's the following in the robots.txt file:

User-agent: (Robot name)
Disallow: www.example.com
Disallow: www.example.com/page1.htm

When the bot sees this it will notate the domain and url, under the directive that this domain and this URL have been dis-avowed by the domain which hosts the robots.txt file. I will leave it up to the search engines to figure out how they want to handle this notated data.

The robots.txt file is advisory information. Shouldn’t we have the ability to further advise the bots when they “read” the robots.txt file?
We advise them on which pages/directories NOT to follow on our site, why not also let us advise them about the links from other domains to our ours?

Realizing this may be like moving mountains to get the robots standards changed, I wanted to bring this idea to this forum for discussion.

This will be well worth the effort for the search engine’s bots, plus it’s a single location for webmaster’s to go to, to make this happen.
1:34 am on Oct 11, 2012 (gmt 0)


WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
votes: 235

the process of "crawling the url database" is completely divorced from "assigning link equity".
robots.txt is about excluding crawlers.
excluding link equity would require a separate protocol.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members