Speed of Googlebot

[Albaba]

is it just me? lately googlebot crawl slowly in my site even some times her sister (freshbot) overlapping.
anyone give me information, the speed of googlebot in your site?

[yetanotheruser]

Hi Albaba,

They claim Googlebot hits once every 15 seconds or so (afaik) ..

Personally watching him going through one of our sites ar about 1 hit per second or so.. two minutes on, half a minute off..

(I think in another thread someone mentioned googlebot's speed being tied to the page response rates?)

HTH :)

[curlykarl]

Is she still munching?

I cant see my logs until midnight?

:)

CK

[Albaba]

Yep she still crawl but lately slow. maybe slammer disturb her

[wingslevel]

I thought about slammer too....

she has only done about 60% as much as she had done last month for the same number of days after the update....

[krbulldog]

Googlebot visited my site(s)about 20 times today..
so she's still spidering...

More active bots today: Inktomi, Alexa, Aol.

[krbulldog]

sorry Aol was a cache-proxy......

[peterdaly]

My experience is google starts slowly on a site. It slow picks up speed as it goes. The longer it crawls a site, the faster it pulls pages. That way it doesn't cause much load on smaller sites, while being able to really whallop the sites that have tons of pages.

I believe I have seen it get up as fast as 1 page per second for long periods of time. I had a site in the Jan crawl that had 50k+ pages indexed. Near the end of the crawl googlebot was slamming against it almost non-stop.

So far this crawl I am at 16.2k indexed pages on a PR5 site. 5.5k+ pages indexed (combined) on another couple of very new sites, with lower PR.

-Pete

[yetanotheruser]

curlykarl, et al,

yup she's still about.. and hungry too..

was showing knowledge of about 12k pages since the last dance, and I'd kinda assume she'd only spider them this month.. but she's taken ~17k in the past couple of days and gaining speed.. :)

.. does that mean that GB's crawl list updates as it goes nowerdays?

haven't seen any other bots though.. (except ai_archiver... does he count? )

:)

[SubZeroGTS]

lol i see that ai_archiver bot too ;)

googlebot has been crawling my site a lot more slowly than last month. my server is also running a bit slow so that probably doesn't help.

i've had way more hits from freshbot (64.68.*) overlapping with deepcrawlbot (216.239.*) though. odd...

[wired4fun]

Regarding deep crawls:

Got lots of hits so far this month, but googlebot has stopped crawling me as of today. I expect (hope) it will be back later today or tomorrow.

For me, googlebot takes the most pages on day 2 or 3 of the crawl, and then slows down from then. It usually
crawls my site for 7-9 days.

[DerekT]

Googlebot has been hitting my server pretty hard this update. So far it has indexed 13K pages in the last 2 days and is still crawling. It has actually taken my server down a couple of times today, requesting numerous pages per second. All the pages are PHP and MySql generated so it generates quite the load on the server. Funny thing is, after the server is brought back online after a stall, Googlebot is requesting pages the instant it is back online. It seems Googlebot is forgiving when it comes to server downs. One of the downs was in the realm of an hour in length.

Anyone else have similar experiences?

Derek