Welcome to WebmasterWorld Guest from 54.167.175.218

Forum Moderators: goodroi

Robots.txt and Index,nofollow

do they have to match

   
3:57 am on Feb 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do the meta tags and the robots.txt file have to match? Here's my example site

index.html
page1.html
page2.html
page3.html
sitemap.html

all pages link to each other. I only want the bots to follow the links from index.html and sitemap.html, but I want the content on all of the pages indexed.

if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?

9:32 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



your waiting for so long for an answer!
No you only put the metatag in your index.html and i advise to put:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
spiders only respect your robot metatag in the index.
you have to allow the "follow" for spider trace the links from your index
10:25 pm on Feb 22, 2004 (gmt 0)

10+ Year Member



To avoid bots to follow page1,2,3 links, the pages that you no whant bots to track have to be in a directory/s disallowd in the robots.txt
Like this:
User-agent: *
Disallow: /my-dir/
In 'my-dir' is where the pages you don't whant to be found.
10:38 pm on Feb 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?


Yes.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month