Googles webcrawler/spider has two modes of operation. One is a "deepcrawler" that indexes the entire web once or twice a month. The second mode is a spider that downloads high ranking or frequently updated pages approximatly every 24 to 72 hours. That daily mode is called "freshbot" to reflect the "fresh" tag Google puts next to recently updated pages in search results. Related Threads:
