Force spider to revisit sooner?

Robots instructions help.

[Fuzzi_Bear]

Is there a command / instruction that can be placed in the robots.txt in oder to instruct spiders to revisit afer x no. of days? - And if so, does this work? (i.e. the reliability of it)

Thanks.

[jdMorgan]

No. The robots decide when (or if) they want to revisit. You can slow some of them down using the semi-proprietary Crawl-Delay: directive, but you can't speed them up.

A bit of a thought experiment will reveal the reason this won't work: If every webmaster requested an hourly re-crawl of his/her entire site, the 'bots could not possibly keep up, and there would be no bandwidth left on the internet for human surfers or e-mail...

Jim

[Dijkgraaf]

Not in the robots.txt, and all other methods such as the REVISIT-AFTER META tag (not support by any major search engine) or Google site map can only "suggest" to the search engine how often to visit a particular page.

I agree with Jim in that you will never be able to force a spider to come and visit as this is just open to abuse.

If you are adding new original content to your site then spiders will be more likely to revisit, so aim for that.

[sonjay]

You should visit the official Robots.txt site [robotstxt.org] for the official scoop on using robots.txt files.