Forum Moderators: open
One site gets crawled all the time, the other less frequently. (Probably worth noting that the one that gets the most crawls is PR7 and the other PR6 - so this is likely to have something to do with this)
They are as follows:
Site 1 (PR7)
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Tue, XX XXX 2004 13:50:24 GMT
Cache-Control: no-store
Connection: Keep-Alive
Content-Length: 21115
Content-Type: text/html
Cache-control: private
Site 2 (PR6)
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Tue, XX XXX 2004 10:50:14 GMT
Cache-Control: no-store
Pragma: no-cache
Content-Type: text/html
Expires: Tue, XX XXX 2004 10:49:15 GMT
Cache-control: no-store
I've been wondering whether any of these differences will have any impact on the way Google spiders the site.
i.e. should I be adding....
Connection: Keep-Alive
Content-Length: 21115
...to site 2
Or is there anything else in there that should be ammended?
Have found this [webmasterworld.com] on "If Modified Since". Applicable to both IIS and Apache?