Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Will spiders follow links on URL's blocked by robots.txt

3:13 pm on Apr 16, 2008 (gmt 0)

New User

10+ Year Member

joined:May 24, 2003
votes: 0

I want to block some URL's using robots.txt but I still want the links on the blocked URL's to be followed. Can I do this or should I just use META NAME="ROBOTS" CONTENT="NOINDEX,"FOLLOW" on the URL's I want blocked?
3:29 pm on Apr 16, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
votes: 0

6:34 pm on Apr 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
votes: 0

The robots noindex meta tag will allow the bot to fetch the page and follow the links from it. They will not list the page in the SERPs.

The robots.txt disallow directive tells the bot to not even fetch the page. Therefore none of the content will be parsed. Note that if there are links to that page, that the URL could still appear as a URL-only entry in the SERPs.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members