Forum Moderators: goodroi

Message Too Old, No Replies

Can I get a page crawled daily?

disallow commands all I can find

         

innocbystr

9:03 pm on Oct 2, 2005 (gmt 0)

10+ Year Member



I have one page on my site that I would like to have crawled daily. Seems like I saw a robots meta tag command for that once upon a time, but now that I could use it all I can find are exclusion commands. I change this page almost daily but it's not my index page so it doesn't get crawled often. Any help would be appreciated.

sem4u

9:25 pm on Oct 2, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is this page linked to your home page? That would help.

innocbystr

9:54 pm on Oct 2, 2005 (gmt 0)

10+ Year Member



Yes, it is linked to my homepage. Most bots just seem to check my robots.txt file, the homepage (can tell this by the file size), and then go on their way. Exceptions are Yahoo's random slurping, then Google and MSN's monthly "sweeps".

Thanks,
Blair

Dijkgraaf

10:57 pm on Oct 2, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Currently there is no way that you can legitemately get a page index daily by all bots.
Possibly a standard will develop like Google's sitemap, in which you can indicate how often a page is likely to change which the bots can then chose to use (or not) as to how often to request the page.

innocbystr

11:33 pm on Oct 2, 2005 (gmt 0)

10+ Year Member



Thanks for the info. Was just kind of hoping there was a way to single out a paticular page for crawling daily, while leaving the others to the "usual crawl pattern". Thought the SE's might like to be alerted to a page with content changing daily as opposed to no change at all.