GoogleBot

Multiday crawl

[willtell]

I have noticed that Google takes quite a few days to crawl our site. Our site does contain around 20k pages, but is this common for GG to hang around for days?

[amoore]

Yep. It looks like it took about 5 days to crawl my site this month and it's a lot smaller than yours.
Here's a breakdown of pages pulled by googlebot each date.

+-------------------+------+ 
¦ date    ¦ hits ¦ 
+-------------------+------+ 
¦ Monday 9/2/02  ¦ 4 ¦ 
¦ Tuesday 9/3/02 ¦ 39 ¦ 
¦ Wednesday 9/4/02 ¦ 49 ¦ 
¦ Thursday 9/5/02 ¦ 46 ¦ 
¦ Friday 9/6/02  ¦ 4 ¦ 
¦ Saturday 9/7/02 ¦ 8 ¦ 
¦ Sunday 9/8/02  ¦ 41 ¦ 
¦ Monday 9/9/02  ¦ 37 ¦ 
¦ Tuesday 9/10/02 ¦ 37 ¦ 
¦ Wednesday 9/11/02 ¦ 63 ¦ 
¦ Thursday 9/12/02 ¦ 36 ¦ 
¦ Friday 9/13/02 ¦ 43 ¦ 
¦ Sunday 9/15/02 ¦ 66 ¦ 
¦ Monday 9/16/02 ¦ 16 ¦ 
¦ Tuesday 9/17/02 ¦ 5 ¦ 
¦ Wednesday 9/18/02 ¦ 66 ¦ 
¦ Thursday 9/19/02 ¦ 34 ¦ 
¦ Friday 9/20/02 ¦ 33 ¦ 
¦ Saturday 9/21/02 ¦ 48 ¦ 
¦ Sunday 9/22/02 ¦ 19 ¦ 
¦ Monday 9/23/02 ¦ 8 ¦ 
¦ Tuesday 9/24/02 ¦ 62 ¦ 
¦ Wednesday 9/25/02 ¦ 16 ¦ 
¦ Thursday 9/26/02 ¦ 2 ¦ 
¦ Friday 9/27/02 ¦ 20 ¦ 
¦ Saturday 9/28/02 ¦ 10 ¦ 
¦ Sunday 9/29/02 ¦ 14 ¦ 
¦ Monday 9/30/02 ¦ 8 ¦ 
¦ Tuesday 10/1/02 ¦ 11 ¦ 
¦ Wednesday 10/2/02 ¦ 110 ¦ <-- start of crawl 
¦ Thursday 10/3/02 ¦ 430 ¦ 
¦ Friday 10/4/02 ¦ 633 ¦ 
¦ Saturday 10/5/02 ¦ 152 ¦ 
¦ Sunday 10/6/02 ¦ 168 ¦ <-- last day of crawl 
¦ Monday 10/7/02 ¦ 49 ¦ 
¦ Tuesday 10/8/02 ¦ 35 ¦ 
¦ Wednesday 10/9/02 ¦ 21 ¦ 
¦ Thursday 10/10/02 ¦ 15 ¦ 
+-------------------+------+ 
 

Sorry, I can't get the table to show up just right.
Anyway, it regularly takes googlebot several days to completely crawl my site