Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Does Googlebot support the If-Modified-Since HTTP header?

         

spiral

9:41 am on Apr 7, 2018 (gmt 0)

10+ Year Member



I saw an old post on the Webmaster Central Blog from 2008 that says they support the If-Modified-Since HTTP header (https://webmasters.googleblog.com/2008/11/date-with-googlebot-part-ii-http-status.html), but can't find anything recent that discusses this. When checking our access logs, I don't see any conditional GETs from Googlebot, but that may be because we don't use Last-Modified tags.

The issue we have is that Googlebot is making a few hundred thousand requests a day for the same JS files.

This is for a large site, but still takes 10% to 15% of our daily crawl budget.

Does anyone here have experience using some kind of cache control tags like above in order to optimize crawling?

phranque

10:29 pm on Apr 7, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



this is the current google line on the subject (from Pagespeed Insights):
Leverage Browser Caching [developers.google.com]

google has been encouraging webmasters to properly handle the If-Modified-Since HTTP Request header at least since Matt Cutts' post here in 2002:
Are you using If Modified Since? [webmasterworld.com]

phranque

10:41 pm on Apr 7, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Does anyone here have experience using some kind of cache control tags like above in order to optimize crawling?

especially if you are on an apache server, this may also be helpful:
Apache Caching Guide [httpd.apache.org]

keyplyr

12:35 am on Apr 8, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



For Apache, I have found the simplest way to set a 30 day cache (the max most browsers support) with the lowest server overhead is:
Header set Cache-Control "max-age=2592000"
The max-age is expressed in seconds. Common max-age values are:
One minute: max-age=60
One hour: max-age=3600
One day: max-age=86400
One week: max-age=604800
One month: max-age=2628000 (31 days)
One year: max-age=31536000 (however most browsers only support 1 month)

phranque

3:00 am on Apr 8, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



my mistake above - i meant to link to this module's doc:
Apache Module mod_expires [httpd.apache.org]

spiral

6:23 am on Apr 9, 2018 (gmt 0)

10+ Year Member



Thanks for the helpful responses :)