I have a client with 2 sites which we've just discovered have differing sets of server headers. One site gets crawled all the time, the other less frequently. (Probably worth noting that the one that gets the most crawls is PR7 and the other PR6 - so this is likely to have something to do with this)
They are as follows:
Site 1 (PR7)
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Tue, XX XXX 2004 13:50:24 GMT
Cache-Control: no-store
Connection: Keep-Alive
Content-Length: 21115
Content-Type: text/html
Cache-control: private
Site 2 (PR6)
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Tue, XX XXX 2004 10:50:14 GMT
Cache-Control: no-store
Pragma: no-cache
Content-Type: text/html
Expires: Tue, XX XXX 2004 10:49:15 GMT
Cache-control: no-store
I've been wondering whether any of these differences will have any impact on the way Google spiders the site.
i.e. should I be adding....
Connection: Keep-Alive
Content-Length: 21115
...to site 2
Or is there anything else in there that should be ammended?
Have found this [webmasterworld.com] on "If Modified Since". Applicable to both IIS and Apache?
