homepage Welcome to WebmasterWorld Guest from 54.227.34.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Will spiders follow links on URL's blocked by robots.txt
deanw

10+ Year Member



 
Msg#: 3628259 posted 3:13 pm on Apr 16, 2008 (gmt 0)

I want to block some URL's using robots.txt but I still want the links on the blocked URL's to be followed. Can I do this or should I just use META NAME="ROBOTS" CONTENT="NOINDEX,"FOLLOW" on the URL's I want blocked?

 

bilalseo

5+ Year Member



 
Msg#: 3628259 posted 3:29 pm on Apr 16, 2008 (gmt 0)

no

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3628259 posted 6:34 pm on Apr 16, 2008 (gmt 0)

The robots noindex meta tag will allow the bot to fetch the page and follow the links from it. They will not list the page in the SERPs.

The robots.txt disallow directive tells the bot to not even fetch the page. Therefore none of the content will be parsed. Note that if there are links to that page, that the URL could still appear as a URL-only entry in the SERPs.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved