Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
They claim Googlebot hits once every 15 seconds or so (afaik) ..
Personally watching him going through one of our sites ar about 1 hit per second or so.. two minutes on, half a minute off..
(I think in another thread someone mentioned googlebot's speed being tied to the page response rates?)
I believe I have seen it get up as fast as 1 page per second for long periods of time. I had a site in the Jan crawl that had 50k+ pages indexed. Near the end of the crawl googlebot was slamming against it almost non-stop.
So far this crawl I am at 16.2k indexed pages on a PR5 site. 5.5k+ pages indexed (combined) on another couple of very new sites, with lower PR.
yup she's still about.. and hungry too..
was showing knowledge of about 12k pages since the last dance, and I'd kinda assume she'd only spider them this month.. but she's taken ~17k in the past couple of days and gaining speed.. :)
.. does that mean that GB's crawl list updates as it goes nowerdays?
haven't seen any other bots though.. (except ai_archiver... does he count? )
Got lots of hits so far this month, but googlebot has stopped crawling me as of today. I expect (hope) it will be back later today or tomorrow.
For me, googlebot takes the most pages on day 2 or 3 of the crawl, and then slows down from then. It usually
crawls my site for 7-9 days.
Anyone else have similar experiences?