Are you using If Modified Since?

You should be!

[GoogleGuy]

I wanted to urge people to configure their server to support the "If Modified Since" header?

Why should you care about "IMS"? When a smart spider like Googlebot comes around, IMS lets you tell the spider that a page hasn't changed. Then Googlebot can use the old copy of the page. That frees up the bot to download more pages while saving bandwidth. Because of the bandwidth savings, IMS hits are almost "free" in terms of server load. Plain apache can serve _lots_ of IMS queries per second before slowing a machine down.

IMS can work for dynamically generated pages too. Someone posted how to do it for PHP-generated pages, for example. The bottom line is that if your server supports IMS correctly, you can tell Googlebot about more pages without as much server load or bandwidth on your part. As Google crawls more often to make the web a fresher place, adding this flag will help you and search engines.

[gopi]

Thanks for the tip GG ...will be useful for sites with large number of pages .

[quiet_man]

Does this tip just apply to those larger enterprises that have their own (dedicated) servers? What about the little guy that only needs / can only afford shared (virtual) hosting? Anything we can do 'client side'? (this is from a non-technie).

[Sasquatch]

quiet_man,

What sort of server is hosting your website?

What language do you use to produce your pages?

It truns out that with most methods of generating dynamic pages, you are able to manually process the headers.

On the other hand, if you are using static HTML, all the settings are server side, and I would hope they are set up properly by default.

[Randex]

"So would this lead to more frequent updates?"

As long as they are not like the latest one.

[Sasquatch]

As long as they are not like the latest one.

I am getting really tired of this topic sneaking into every thread. Please refrain from adding these comments every time someone mentions "update".

[GoogleGuy]

"If you'll be saving bandwidth you can do your crawl faster right?"

Right on, Chico_Loco. The other nice bonus is that if we don't have to waste time fetching pages that haven't changed, that gives us a chance to crawl more new pages. Everybody wins because crawling gets more efficient.

[stuntdubl]

Good tip Google Guy.....
It's nice when a suggestion is mutually beneficial:)
It keeps from sweet lil' googlebot hammering servers too!:)

I just checked on Brett's tool, and it said that I had it turned on. I'm not REAL techie, but my boss and I just installed this Linux (apache) webserver recently......is 'last modified' turned on by default?

[john5]

Hi GoogleGuy

My web site is not hosted on an Apache server but on Microsoft IIS. Anything like this "If Modified Since" available for the Microsoft NT/IIS platform?

[gattrill]

john5 - you can read headers in ASP on IIS by using the Request object. I am not in a position to test it, but I think 'Request.ServerVariables("HTTP_IF-MODIFIED-SINCE")' would return the date GoogleBot is asking for you to check.

You would then check that internally and if your page has not changed return 'Response.Status="304 Not Modified"' and 'Response.End'. If it has changed simply return the page as normal.

Gareth

[rfgdxm1]

ME! Does this mean Googleguy I'll get higher SERPs because the server I'm on does so? ;) While I don't pay that much attention to the raw logs, I definitely know that the server gives 304 responses. I noticed Googlebot is nosy and comes around like every 2 days for the home page, and is getting a 304.

[Finder]

Someone posted how to do it for PHP-generated pages

Does anyone know which thread this refers to? I did a couple of searches and couldn't find anything.

My server does send 304 responses, but only for image files. My index page has PHP at the very top that creates an "expires" header each time the page is requested. But there is no if-modified-since header being sent with that page.

[Sasquatch]

The solution was incomplete since he did not send the 304 if the dates were the same.

Do a search on "php if modified since" without quotes in google and you should find some good sites right there on the front page.

[athinktank]

My server is returning a correct 304 for pages that are not updated. However, Googlebot gets the 304 and then immediately requests the page again and gets a 200 and the full payload. Is google currently asking for the page no matter what? Or is this a new feature that we will see soon. Or is there something more to this than returning a 304.

[nancyb]

This is a timely thread for me, I think, but I could just be totally confused too. Is this the same as TTL (time to live)?

I just moved my site from one host to another (Sunday about noon the DNS name server was changed) Late yesterday I started seeing hits in the new server logs including bunches of visits from ms. googlebot today :) - one worry down.

However, I checked the old server logs a few minutes ago and there are just a few hits there now - most are my own IP and some from inktomi slurp :(

My question .. am I still seeing my own hits in the old log because my ISP has not updated their DNS? I've dumped the cache manually several times today. I know I'm still seing the old server files because I used absolute urls there, but on the new server I'm using relative urls. Can't get my email downloaded from the new server either and I'm assuming this is an ISP DNS issue.

My old server logs have consistently shown 304's when I changed pages so I'm not sure if this discussion is the same thing as TTL or not.

Thanks..

<edit-- said that wrong, 304 when the file was not changed >

[martin]

GoogleGuy tell Googlebot to send Accept-Encoding: gzip, deflate, x-gzip and you can save even more bandwidth.