Content Change and Googlebot Behavior

Measuring Content Change to index rate

[lstrand]

Has anyone effectively measured rates of content change and the effects they have on Google's crawls and crawl to index ratios? I am attempting to build a measurement test for this metric, if anyone else has any insight or ideas it would be much appreciated.

[Quadrille]

There's a bunch of variable to juggle, including what the search term was and how related the changes are to that term.

You'll need to be able to assess how 'competitive' your test search terms are before you start - and you'll probably have to make judgements about why competive term changes get smaller serp changes as a result.

Plus you'll need to allow for changes to competitor sites, and you'll need to do the research on specific known datacenters, for continuity.

You'll need a stable site that's been there for at least 18 months, stable for at least three. And you'll need to allow 4-6 weeks (for a complete spidering cycle) before assessing each change.

Good Luck; I suspect you'll need it! :)

[edited by: Quadrille at 9:02 pm (utc) on Sep. 26, 2006]

[lstrand]

Essentially my plan is initially to isolate the value of a complete spider cycle. Until that is known, it is pure speculation. To do that I am using historical crawl data on one site of over 24 months looking for trending in crawl activity. (I chose 24 months to give ample pre big-daddy and potentially measure how much Google has altered their crawl activity with BD update) I also have historical index data and have already calculated ~.029% to be an average number of index/ reindex. The content is stagnant, it is purely product data that does not change, consequently, it is hard to find fresh content. Nonetheless I intend to use press releases and company news (all relevant to each particular product segment) as fresh content and attempt to determine how much new content constitutes "new" in the eyes of G. Implementing this new content in various quantities of new text over each cycle I will revisit the index to crawl ratio and evaluate its change. Q- did I miss anything you think?

Best Regards