Forum Moderators: open
Why are some sites updated frequently and others left to continue displaying old information?
Why are some sites updated frequently and others left to continue displaying old information?
Sites with lower PageRank tend not to be crawled as frequently as those with higher PR.
Last modified dates being returned in the server header. If there are no updates, then there is no reason for Freshbot to linger around.
what is the usual timescale from freshbot spidering to indexing?
Thank you for your reply.
I am comparing my site (post April) with two other sites I have designed (pre April) which are updated daily. Both these sites have a PR4, the same as mine.
My site has fresh info daily. The software is so designed that all the Country pages in the left hand navigation panel are refreshed daily.
I forgot to mention that to illustrate how up to date the pages are, click on 'France' in the left hand panel of the index page and look at the date of each property advert.
It seems to compare the last modified date with a value it holds from several weeks previous, so many sites are freshed again regardless of whether they were actually modified since the bots last visit.
As to why some sites are freshed more than others I believe it has to do with the total number of incoming links and where those links come from. In most cases this will correlate somewhat to the PR of the site, hence it seems PR4 to PR5 sites and above get freshed a lot.
But, take 2 PR4/5 sites that get their PR in difference ways and the odds of the bot finding you can be very different.....hence you may not get freshed so often as a site that appears to be equal.
Just got ......."Thank you for your note. Google's robots crawl the web on a regular cycle of four to six weeks, indexing more than three billion web pages. Because of the vast amount of data refreshed with each crawl, the update process is completely automated. This is the standard crawl and schedule for Google's robots. Any other updates or crawls are merely experiments on Google's part to find methods of improving our search"
That sure explain why three similar sites get updated differently.
