homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Speed of Googlebot

 2:06 pm on Feb 10, 2003 (gmt 0)

is it just me? lately googlebot crawl slowly in my site even some times her sister (freshbot) overlapping.
anyone give me information, the speed of googlebot in your site?



 2:15 pm on Feb 10, 2003 (gmt 0)

Hi Albaba,

They claim Googlebot hits once every 15 seconds or so (afaik) ..

Personally watching him going through one of our sites ar about 1 hit per second or so.. two minutes on, half a minute off..

(I think in another thread someone mentioned googlebot's speed being tied to the page response rates?)

HTH :)


 3:03 pm on Feb 10, 2003 (gmt 0)

Is she still munching?

I cant see my logs until midnight?




 3:19 pm on Feb 10, 2003 (gmt 0)

Yep she still crawl but lately slow. maybe slammer disturb her


 4:00 pm on Feb 10, 2003 (gmt 0)

I thought about slammer too....

she has only done about 60% as much as she had done last month for the same number of days after the update....


 4:09 pm on Feb 10, 2003 (gmt 0)

Googlebot visited my site(s)about 20 times today..
so she's still spidering...

More active bots today: Inktomi, Alexa, Aol.


 4:10 pm on Feb 10, 2003 (gmt 0)

sorry Aol was a cache-proxy......


 4:58 pm on Feb 10, 2003 (gmt 0)

My experience is google starts slowly on a site. It slow picks up speed as it goes. The longer it crawls a site, the faster it pulls pages. That way it doesn't cause much load on smaller sites, while being able to really whallop the sites that have tons of pages.

I believe I have seen it get up as fast as 1 page per second for long periods of time. I had a site in the Jan crawl that had 50k+ pages indexed. Near the end of the crawl googlebot was slamming against it almost non-stop.

So far this crawl I am at 16.2k indexed pages on a PR5 site. 5.5k+ pages indexed (combined) on another couple of very new sites, with lower PR.



 6:09 pm on Feb 10, 2003 (gmt 0)

curlykarl, et al,

yup she's still about.. and hungry too..

was showing knowledge of about 12k pages since the last dance, and I'd kinda assume she'd only spider them this month.. but she's taken ~17k in the past couple of days and gaining speed.. :)

.. does that mean that GB's crawl list updates as it goes nowerdays?

haven't seen any other bots though.. (except ai_archiver... does he count? )



 6:15 pm on Feb 10, 2003 (gmt 0)

lol i see that ai_archiver bot too ;)

googlebot has been crawling my site a lot more slowly than last month. my server is also running a bit slow so that probably doesn't help.

i've had way more hits from freshbot (64.68.*) overlapping with deepcrawlbot (216.239.*) though. odd...


 7:17 pm on Feb 10, 2003 (gmt 0)

Regarding deep crawls:

Got lots of hits so far this month, but googlebot has stopped crawling me as of today. I expect (hope) it will be back later today or tomorrow.

For me, googlebot takes the most pages on day 2 or 3 of the crawl, and then slows down from then. It usually
crawls my site for 7-9 days.


 9:04 pm on Feb 10, 2003 (gmt 0)

Googlebot has been hitting my server pretty hard this update. So far it has indexed 13K pages in the last 2 days and is still crawling. It has actually taken my server down a couple of times today, requesting numerous pages per second. All the pages are PHP and MySql generated so it generates quite the load on the server. Funny thing is, after the server is brought back online after a stall, Googlebot is requesting pages the instant it is back online. It seems Googlebot is forgiving when it comes to server downs. One of the downs was in the realm of an hour in length.

Anyone else have similar experiences?


Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved