Forum Moderators: open
I'm not quite sure I understand the question. Are you looking for info on Google's deep crawling behavior?
The first is the crawler responsible for finding and indexing new content on the web. This is the one that is often referred to as the deep crawler. It will show up on your site from an IP address in the 216.239.46* range.
The deep crawler is very aggressive and it will attempt to crawl the majority of content on your site. Its primary time of operation is during the first two weeks of the month. The next deep crawl will probably begin around Monday, and the pages collected in that crawl will begin to show up in the first week of January.
The second type of crawler is what is comonly referred to as the Fresh Bot. This bot visits selected sites on a daily basis and indexes pages that are new or have been changed since the last deep crawl.
The new pages the Fresh Bot picks up are added to the database almost instantly and they tend to not stick around for too long. But they due tend to rank well while they are there.
The combination of the two bots cause the google search results to change quite regularly. You can find much more information on that process by searching for the term Everflux.
Craig
