Googlebot 2.1 strange behavior on deep crawl

strange deep crawl behavior by Googlebot

[stevens]

About 7 weeks ago I added 70 new pages to my site, and waited for the Googlebot deep crawl. Well, a few days ago it arrived, and over the course of 24 hours Googlebot hit every one of my old pages. However, not a single one of the 70 new pages was hit. I find this to be very odd. I should mention that all my old and new pages are in the top end of the diretory, and the new pages were created from the same template as the old ones. Only the text on the page, the metatags and the title were altered on the new pages. Has anyone experienced this type of behavior from Googlebot? I can't think of a reason why it would hit every old page on my site on a deep crawl, yet skip 70 new pages. Thanks.

[stevens]

Ahem. Any thoughts on this?

[irishaff]

Hi,

It may be that the bot is going directly to URLs it has cached for your site and not spidering down from the homepage. Im sure you have checked robots.txt etc . to see that your not blocking the new section etc.

All being equal give it some time.

David

[jdMorgan]

Yes, ahem, it's the holidays for many members... Gets kinda quiet around here sometimes.

This behaviour would be typical of Googlebot checking only the pages it already knows about for updates. It may note the new links and come back for them later, or perhaps a separate 'bot will visit with the express purpose of looking for new links and pages.

A few years ago, we called the behaviour you observe the "Freshbot" function, where Google was looking for updates only on pages that it had previously indexed. This was a daily-to-weekly function, depending on the PageRank of the pages. Then, approximately once a month, we'd see the "Deepbot" which would spider almost every page it could find. So, Freshbot was intended to keep the cache and index of previously-indexed pages fresh, and Deepbot was used less frequently to discover new pages. It's quite possible that we're seeing a return to this behavioural pattern.

The solution? Wait up to 90 days and then check again. If the new pages aren't showing up by then, you can be sure something's wrong.

Jim

[stevens]

Thanks for the feedback. I've had the site running for two years now and I've never experienced this. Yes, my robots.txt file is in order. Jim may be on to something regarding the return to "freshbot" behavior. In any event, the old standby "behavour" for webmasters dealing with Google is probably my only recourse right now: patience.

[BillyS]

My experience with googlebot based on a site with a PR5.

Googlebot crawls old pages (at least a month old) about 4 time each month and deep crawls everything about once a month. In retrospect, these heavy crawl sessions usually happen several days before and after an update.

[stevens]

My experience is based on a site with a PR 6.

I have always been deep crawled about once a month until this most recent problem I've alluded to.

However, I'm wondering if the recent increase in the size of the Google database has something to do with my site being deep crawled less often.

Essentially, the number of SERPs has more than doubled for my keywords. Maybe Google hasn't quite doubled the number of bots.