Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
They claim Googlebot hits once every 15 seconds or so (afaik) ..
Personally watching him going through one of our sites ar about 1 hit per second or so.. two minutes on, half a minute off..
(I think in another thread someone mentioned googlebot's speed being tied to the page response rates?)
I believe I have seen it get up as fast as 1 page per second for long periods of time. I had a site in the Jan crawl that had 50k+ pages indexed. Near the end of the crawl googlebot was slamming against it almost non-stop.
So far this crawl I am at 16.2k indexed pages on a PR5 site. 5.5k+ pages indexed (combined) on another couple of very new sites, with lower PR.
yup she's still about.. and hungry too..
was showing knowledge of about 12k pages since the last dance, and I'd kinda assume she'd only spider them this month.. but she's taken ~17k in the past couple of days and gaining speed.. :)
.. does that mean that GB's crawl list updates as it goes nowerdays?
haven't seen any other bots though.. (except ai_archiver... does he count? )
googlebot has been crawling my site a lot more slowly than last month. my server is also running a bit slow so that probably doesn't help.
i've had way more hits from freshbot (64.68.*) overlapping with deepcrawlbot (216.239.*) though. odd...
Got lots of hits so far this month, but googlebot has stopped crawling me as of today. I expect (hope) it will be back later today or tomorrow.
For me, googlebot takes the most pages on day 2 or 3 of the crawl, and then slows down from then. It usually
crawls my site for 7-9 days.
Anyone else have similar experiences?