Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and Index,nofollow

do they have to match

         

graywolf

3:57 am on Feb 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Do the meta tags and the robots.txt file have to match? Here's my example site

index.html
page1.html
page2.html
page3.html
sitemap.html

all pages link to each other. I only want the bots to follow the links from index.html and sitemap.html, but I want the content on all of the pages indexed.

if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?

Uzil

9:32 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



your waiting for so long for an answer!
No you only put the metatag in your index.html and i advise to put:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
spiders only respect your robot metatag in the index.
you have to allow the "follow" for spider trace the links from your index

Uzil

10:25 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



To avoid bots to follow page1,2,3 links, the pages that you no whant bots to track have to be in a directory/s disallowd in the robots.txt
Like this:
User-agent: *
Disallow: /my-dir/
In 'my-dir' is where the pages you don't whant to be found.

DaveAtIFG

10:38 pm on Feb 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?


Yes.