Google Inclusion Cycle

When Does Google Crawl It's Seeds?

[Napoleon]

More interesting for me than when Google will re-index is: when will it crawl Yahoo, ODP and other seeds for inclusion in the new database.

The new index will happen... it is beyond my control and its contents will reflect the past... what has already happened. Interesting, but probably at this point well beyond our control.

Within our control, however, is when to submit to the seeds. Must this be early in the Googlebot crawl cycle, or can we get away with late?

I suggest that it is very early for ranking contribution purposes, but late for just plain inclusion in the index. Do others share this experience?

Another issue is whether the seeds are crawled at the same stage. Will Yahoo be covered earlier or later than ODP? My view is that ODP is later, but I have not performed substantial research... and of course this may change.

What of other seeds? Which generic directories are important and when are they actually crawled?

This of course is not solely an issue of timing - it is necessary for analysis. It is very hard to analyze Google at the best of times, but when you cannot be sure which seeds are fully factored it is almost impossible.

Any firm experiences here?

[Grumpus]

I have several sites in DMOZ and the time for inclusion into the Google directory seems roughly equal to how long it takes to get into google's main index every time. If I created a page today, (barring freshness) it'll likely show up in Google's index at the end of November. If it gets into the DMOZ today, it'll likely show up at the end of November. If it gets into the DMOZ late into the normal deep crawl cycle and beyond, it doesn't usually show til December's update (unless I'm lucky and that cat hasn't gotten crawled).

I can't say this with 100% certainty, but I'd say it's "pretty firm". Interestingly, the PR benefits from listing at the DMOZ seem to take a full 30 days longer (1 update) to propogate than the listing itself.

G.

[ciml]

Napoleon:
> I suggest that it is very early for ranking contribution purposes, but late for just plain inclusion in the index. Do others share this experience?

I would very much agree with that, if you have at least one link from a high-ish PR page.

It seems as though Google will visit a reasonable PR page several times in the cycle, but only credit its links to other pages if they're there at the start of the cycle.