Welcome to WebmasterWorld Guest from 54.145.221.99

Forum Moderators: goodroi

Message Too Old, No Replies

Revisit instruction for robots

I want to specify different revisit instruction different sections

   
11:01 am on Jan 9, 2004 (gmt 0)

10+ Year Member



We have a portal with more than 1,00,000 html pages. Almost all sections are crawled by robots. However, all pages are not crawled and hence, many of the pages are leftout. Hence, I want to give different revisit instruction for different section so that over a period of time all pages are crawled. I am not very knowledgeable for robot.txt. Can anybody help me for <b>revisit instruction</B> in robots.txt or anything which ensures <b>all pages are indexed</b>.
11:05 am on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google, Inktomi and ATW will eventually crawl all pages if they have backlinks.

No tricks with robots.txt needed here, just hand tight and eventually they will be crawled by the major players :)

5:13 am on Jan 10, 2004 (gmt 0)

10+ Year Member



Thanks. However, I have observed that some pages appear and then after sometimes they appear. Because of this at any time hardly 25% to 40% of the pages are fatched. Can you advise me how to overcome this.