GoogleBot daily crawling revisited

Another theory as to why some sites get crawled daily

[bodine]

I remember reading here a few months ago that some sites get crawled daily (or almost daily) by Googlebot. Some have thought that it was because of the high PR of the site. But, many have pointed out that the sites may or may not have a high PR. In fact, one instance is one where the site had a PR0:

[webmasterworld.com ]

(You can search for others, as there are many sites that crawled daily.)

I have another theory as to why this may happen. It does not have to do with the PR of the site itself, but with the PR of the sites that original site links to.

For an example, I run MySite.com. I have a link to LinkedTo.com. LinkedTo.com has a high PR because many people link to it, including me. MySite.com does not have as high a PR as LinkedTo.com, but MySite.com is the one that gets crawled daily. Why would that be?

Well, my theory involves one of the fundamental problems I see with using PageRank (which is how I came to this conclusion)-- that once pages are created, they sometimes never get updated, yet they still pass on PR to the linked pages. We all probably have a few pages that we have made, got tired of and never got around to updating again. Any links we have to other sites will still get some PR from us, but because the site is old, the links may have changed, or died, or are no longer relevant.

One of the things Google wants to avoid is having pages at the top of their results that are old. So, it stands to reason that webmasters who update their sites frequently probably check their links more often than those who have not touched their pages in a while, right? So, if you update your site, the PR that you pass on is better than a page that has not been updated in a while. Think of it as decaying PR-- i.e., over time, if your site has not been updated, you will pass along less PR than if you had been updating the site frequently. Google needs to know how often you update your site, so it crawls your site often. [I hope I have explained this well enough for others to understand. If not, ask me to clarify.]

Therefore, it is better to have links from sites that get updated more frequently (blogs, for instance) than a page that gets updated once a year (like a buried page of links).

I'm sure there are other factors involved as to why pages get crawled daily, but I would like comments on this theory. Anyone with a daily crawl schedule want to check some of their outgoing links and report back? I'm interested in hearing the results...

[Visit Thailand]

Interesting thoery, and I think the daily crawl is not per site but per pages. We have one site that has numerous pages which are updated daily with the date tag in the search but not all the pages even though they have been updated.

[vitaplease]

bodine,

I would say that a PR0 page that gets frequent fresh crawls, would have to do with the fact that the page did get recent new links towards it.

[webmasterworld.com...] messages 128/132

I have had a feedback page with a daily crawl date stamp for a while. (why?, probably because I recently added a lot of new pages that all had a link to that feedback page.

Another example is the profile pages of moderators in this forum. Every once a while they get a frequent crawl/caching.

[webmasterworld.com...]

[BryantStevens]

Oops. Wrong reply to wrong message! Please disregard!

[Slud]

I've had two sites start getting crawled daily as soon as they were accepted in to Yahoo.