homepage Welcome to WebmasterWorld Guest from 54.205.99.71
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
how to let robot crawl the links at a page but not index this page?
index, robot
johnlim




msg:1525256
 12:13 am on Jan 26, 2004 (gmt 0)

But I want google to crawl the links at this page. Only don't want google to index or archive this page.

For exapmle, this page is siteA.com/pageA.html and it include the links pageA1.html and pageA2.html, will these two pages pageA1.html and pageA2.html be crawled and archive, indexed by google?

(there will not put any <meta name=robots value=noindex> in these two pages pageA1.html and pageA2.html)

Thanks,

 

ThomasB




msg:1525257
 11:39 pm on Jan 26, 2004 (gmt 0)

"noindex,follow" should be what you're looking for
[robotstxt.org...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved