Forum Moderators: open
What I'm thinking is that googlebot visits, sees that pages have not changed and then gets the heck out of there. However, if pages have changed, then a deep crawl required flag is set and performed later.
If this is true then is it a good idea to 'refresh' all webpages in a site. This is a simple case of like, changing the name of a .gif file which is on each page, or removing a trailing space after <html> etc. You then upload the complete site and to the crawlers, it looks as if your content has changed because of date stamp and file size.
In my case, Google is giving that honour to the two highest pagerank pages of my site (index PR6 and menu-type page PR5). These two pages tend not to be the most fresh or most changed or updated pages of my site at all.
For the rest a high pagerank tends to help most for deep crawling.
Even better, get the internal pages some high quality inbound external links.
Fresh stamp, as from my experience, seems to be tied up with PR6 and above.
-updates are limited only/mostly to the index page, sometimes to the next level directory, it doesn't go further than that, regardless if you have new contents in your inner pages.
-if Google deemed that your site deserves to be visited daily, it will grab your home page regardless of timestamp(ie, news sites - where partial contents changes dynamically using SSI, yet page timestamp remains the same.
I've now stopped changing it and I'm hardly ever fresh. So I think that timestamp may be a factor.
But the main point is that you will not force a full crawl of your site by making slight changes.
And, the really stange thing - the whole time I was at PR0 (Dec-Apr) I was 'freshed' whenever I changed anything on the home page. As far as I can see, by my own site and reading about others' fresh tag, there doesn't seem to be a standard for the 'fresh' although I'm sure there is one.
The Fresh listings follow Open Directory and Yahoo! entries, even if the link is quite deep and the page has only a little PR.
Ciml, thats interesting, are you sure thats the case for every ODP/Yahoo listing?
Have you seen that with sites with say five different pages all listed in DMOZ and all five pages getting the Fresh stamp? (hard to check at this moment because the date stamps often alterate to be presented every other day).
It would be yet another case of Google taking sides with ODP and to a limited extent Yahoo.
If you are running dynamic content, then there won't be any LM header. And in therory, the page will still get delivered to Googlebot when it makes an "If Modified request."
However, I've found that the spidering seems to improve a bit if you actually send a date.
This does not mean that they get date stamps, though. One covers a period from 1302AD to 1329AD, so it doesn't need to get updated often;).
<added>I've just noticed; that page does get the date stamp, while others that have last modified headers are crawled daily but don't get the date stamp. (It just happens that the site I mentioned has SSI without the x-bit-hack. Bad Calum...)</added>
