Welcome to WebmasterWorld Guest from 54.162.44.105

Forum Moderators: goodroi

Message Too Old, No Replies

Revisit instruction for robots

I want to specify different revisit instruction different sections

     
11:01 am on Jan 9, 2004 (gmt 0)

New User

10+ Year Member

joined:Jan 9, 2004
posts:2
votes: 0


We have a portal with more than 1,00,000 html pages. Almost all sections are crawled by robots. However, all pages are not crawled and hence, many of the pages are leftout. Hence, I want to give different revisit instruction for different section so that over a period of time all pages are crawled. I am not very knowledgeable for robot.txt. Can anybody help me for <b>revisit instruction</B> in robots.txt or anything which ensures <b>all pages are indexed</b>.
11:05 am on Jan 9, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 1, 2002
posts:1580
votes: 0


Google, Inktomi and ATW will eventually crawl all pages if they have backlinks.

No tricks with robots.txt needed here, just hand tight and eventually they will be crawled by the major players :)

5:13 am on Jan 10, 2004 (gmt 0)

New User

10+ Year Member

joined:Jan 9, 2004
posts:2
votes: 0


Thanks. However, I have observed that some pages appear and then after sometimes they appear. Because of this at any time hardly 25% to 40% of the pages are fatched. Can you advise me how to overcome this.