Forum Moderators: goodroi
I would like to know if a page blocked by a robot.txt will see his links followed by the spider, in other words, is the robots.txt equal to the syntax <meta name="ROBOTS" content="NOINDEX,FOLLOW">?
Thanks in advance for any feedback,
Regards,
David Camus
The robots.txt exclusion says to not fetch the page at all. Good bots should obey that (bad bots ignore it and other methods are needed to stop them accessing the page).
However, if there are links to that page from some other site, then Google can still show that page as a URL-only entry in the SERPs.
Yahoo goes one further for URL-only entries and "invents" a title using the anchor text of one of the incoming links.
If you don't want even the URL in the SERPs then you instead need a <meta name="robots" robots="noindex"> tag on the page so that the page is fetched and the bot is then told to not index it all. In that case you must not exlude the URL in robots.txt.
.
robots.txt is all about the bot not fetching the page. The page can still appear as a URL-entry.
meta robots is all about not showing results in the SERPs. The page is still fetched by the bot.
Make sure that you use the right one. Do NOT use both for any specific URL.
