Googlebot Crawling Large Sites Less?

July 14th change in Googlebot behaviour.

[danny]

Until July 14th, Googlebot used to fetch every page on my site daily - or more often, as there were typically 1400 or so visits a day and my site has about 1000 pages.

Since July 14th, there have been almost exactly 100 visits from Googlebot a day - between 92 and 106, to be precise, but it looks suspiciously like 100/day with some noise.

Has anyone else noticed anything like this?

[nalin]

I have seen similar behaviour over the exact timeframe you mention with two sites total. In general these sites were spidered for ~1k pages a day which has dropped to roughly the levels you mention with the exception of a relatively complete spidering corresponding to a change in the footer of one of the sites.

My take on this is that the sites in question seem to change often but really do not (there are for instance random products listed on each pages which would likly be different with two GETs but the pattern would be appearent over a hundred spiderings). Perhaps google has tuned their filters to necessitate new content truly be new.

The other thing which I have yet found time to look at is googlebots name recently changed and my cron script did not...I am unsure whether I am indeed getting log content for both googlebots or just the older style one.

[diamondgrl]

haven't noticed anything. my google crawls vary widely, from 2000 to 20000 pages a day, so it's not easy to notice a trend over a few short days but i certainly have had some big crawls in the past week.

[VigoZ]

i think than the numbers of pages crawled by googlebot may change with the change of your pagerank

sites with high paerank are crawled more than sites with less pagerank, please give us your site pagerank, this will help to make this post more usefull to everyone

Thanks.

[danny]

False alarm, sorry guys -- Googlebot just changed its agent string and my grep regexp stopped matching it!

Googlebot is still hitting my 1000 page site 1400 times a day.