| Will spiders follow links on URL's blocked by robots.txt
|
deanw
msg:3628261 | 3:13 pm on Apr 16, 2008 (gmt 0) |
I want to block some URL's using robots.txt but I still want the links on the blocked URL's to be followed. Can I do this or should I just use META NAME="ROBOTS" CONTENT="NOINDEX,"FOLLOW" on the URL's I want blocked?
|
bilalseo
msg:3628282 | 3:29 pm on Apr 16, 2008 (gmt 0) |
no
|
g1smd
msg:3628474 | 6:34 pm on Apr 16, 2008 (gmt 0) |
The robots noindex meta tag will allow the bot to fetch the page and follow the links from it. They will not list the page in the SERPs. The robots.txt disallow directive tells the bot to not even fetch the page. Therefore none of the content will be parsed. Note that if there are links to that page, that the URL could still appear as a URL-only entry in the SERPs.
|
|
|
