homepage Welcome to WebmasterWorld Guest from 54.167.11.16
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Revisit instruction for robots
I want to specify different revisit instruction different sections
adityamodi




msg:1526719
 11:01 am on Jan 9, 2004 (gmt 0)

We have a portal with more than 1,00,000 html pages. Almost all sections are crawled by robots. However, all pages are not crawled and hence, many of the pages are leftout. Hence, I want to give different revisit instruction for different section so that over a period of time all pages are crawled. I am not very knowledgeable for robot.txt. Can anybody help me for <b>revisit instruction</B> in robots.txt or anything which ensures <b>all pages are indexed</b>.

 

percentages




msg:1526720
 11:05 am on Jan 9, 2004 (gmt 0)

Google, Inktomi and ATW will eventually crawl all pages if they have backlinks.

No tricks with robots.txt needed here, just hand tight and eventually they will be crawled by the major players :)

adityamodi




msg:1526721
 5:13 am on Jan 10, 2004 (gmt 0)

Thanks. However, I have observed that some pages appear and then after sometimes they appear. Because of this at any time hardly 25% to 40% of the pages are fatched. Can you advise me how to overcome this.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved