homepage Welcome to WebmasterWorld Guest from 54.227.11.45
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt and Index,nofollow
do they have to match
graywolf




msg:1528543
 3:57 am on Feb 13, 2004 (gmt 0)

Do the meta tags and the robots.txt file have to match? Here's my example site

index.html
page1.html
page2.html
page3.html
sitemap.html

all pages link to each other. I only want the bots to follow the links from index.html and sitemap.html, but I want the content on all of the pages indexed.

if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?

 

Uzil




msg:1528544
 9:32 pm on Feb 22, 2004 (gmt 0)

your waiting for so long for an answer!
No you only put the metatag in your index.html and i advise to put:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
spiders only respect your robot metatag in the index.
you have to allow the "follow" for spider trace the links from your index

Uzil




msg:1528545
 10:25 pm on Feb 22, 2004 (gmt 0)

To avoid bots to follow page1,2,3 links, the pages that you no whant bots to track have to be in a directory/s disallowd in the robots.txt
Like this:
User-agent: *
Disallow: /my-dir/
In 'my-dir' is where the pages you don't whant to be found.

DaveAtIFG




msg:1528546
 10:38 pm on Feb 22, 2004 (gmt 0)

if I put this tag

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

on page1.html, page2.html, page3.html will that work the way I intend?


Yes.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved