Welcome to WebmasterWorld Guest from 54.196.144.100

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and Index,nofollow

do they have to match

     

graywolf

3:57 am on Feb 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do the meta tags and the robots.txt file have to match? Here's my example site

index.html
page1.html
page2.html
page3.html
sitemap.html

all pages link to each other. I only want the bots to follow the links from index.html and sitemap.html, but I want the content on all of the pages indexed.

if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?

Uzil

9:32 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



your waiting for so long for an answer!
No you only put the metatag in your index.html and i advise to put:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
spiders only respect your robot metatag in the index.
you have to allow the "follow" for spider trace the links from your index

Uzil

10:25 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



To avoid bots to follow page1,2,3 links, the pages that you no whant bots to track have to be in a directory/s disallowd in the robots.txt
Like this:
User-agent: *
Disallow: /my-dir/
In 'my-dir' is where the pages you don't whant to be found.

DaveAtIFG

10:38 pm on Feb 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?


Yes.