If I have a folder that I've excluded using the robots.txt file, and I have a link on my site going into that folder, will the robots stay out, or do I need to use a nofollow tag on the link?
Thanks.
jay5r
11:13 pm on Jul 31, 2006 (gmt 0)
Well-behaved robots will stay out, but you may see the link in the SE index, but you'll notice it's just a URL with no cached version available. That's because they're aware of the URL, but know not to retrieve it. Nofollow might keep the URL out of the index, haven't tried it, so I'm not sure.
JasperSEO
1:39 am on Aug 1, 2006 (gmt 0)
Thanks. Anyone else tried this with nofollow and know one way or the other?