Using the expires header to control crawl frequency

[thuss]

I work at a large site that is heavily crawled. The problem is some pages get crawled multiple times per day (which is a costly waste of bandwidth) while other pages almost never get crawled.

Do Googlebot and Slurp pay attention to the Expires (or perhaps another http header) so that we can tell it that the page expires in a week so it won't crawl it multiple times per day?

My hope is that by slowing down the crawl on unnecessarily heavily hit pages we'll speed up the crawl on less frequently visited pages.

Thanks for any advice!

[caveman]

No.

Wouldn't it be nice, though, if we had that sort of control over when, how, and how frequently they crawl? ;-)

[johnblack]

Do you use Google a sitemap? In a Google sitemap you define some sort frequency for each page on your site and that, in theory, affects the frequency of the Googlebot's crawling.

It won't solve all of your bot problems and I'm not sure how effective it is but may be worth a shot if bots sucking bandwidth is a real issue for you.