I've long wondered why Google seemed to prefer static, .htm/.html pages. Recently, after browsing the Google webmaster guidelines and spotting something I'd missed a thousand times before, my eyes have been opened to the 'last-modified' HTTP header issue.
After some lengthy reading, I discovered the HTTP headers of my ASP (classic) pages were lacking in one salient piece of information - the 'last modified' date. This is a side-effect of using server-side scripting to dynamically produce pages. 'Flat' files (.htm, .html, .css, .jpg, .gif etc.) are automagically delivered with the approriate 'last modified' headers by the server - thankfully.
Having adapted my scripts to correctly produce a 'last modified' header for each of my pages, I'm happy that the server headers are correctly reporting the last time I updated the page.
In addition, should a client be checking the last modified date for the purposes of caching (using an If-Modified-Since header), the dates are compared and 304 code is returned when appropriate.
Given that this isn't entirely a simple problem to solve and that Google might see a significant benefit to the operation of their systems if they had access to this info for *every* page they indexed, I have summized that Google developed their 'Sitemaps' (at least in part) to get webmasters to provide this valuable information in another format.
However, now that I am confident I have restored this functionality to my pages, I'm seeing my Google Sitemap as a 'stale' file, with out-of-date 'last modified' info for my pages, since I don't tend to have time to update it every time I update one of my .asp files.
Am I right in thinking that I can ditch my Google Sitemap now as it serves simply as a 'fix' for a problem that I have now solved?
[edited by: Panic_Man at 10:17 am (utc) on Aug. 5, 2007]