Forum Moderators: open

Message Too Old, No Replies

Are the Googlebots slowing down as they mature?

         

uber_boy

2:42 pm on Mar 17, 2003 (gmt 0)

10+ Year Member



After waiting a couple of days, the Googelbots are now hitting my site with some intensity. My site's got a lot of content (> 3m pages) and, thus, for me the big issue is how many pages get read during the Deep Crawl. That said, there was a time not that long ago when the Googlebots would crawl as many as two pages per second (for two or more days). Now, however, they seldom read in more than a page every two seconds (for four or five days). And while it's possible that Google's just less interested in my site, I can't help but wonder whether the Googlebots have become kinder, gentler creatures as they've matured. Thus, in pursuit of an answer, I'd be interested to know just how hard the Googlebots are hitting others' sites, particularly those with a lot of content.

skipfactor

3:31 pm on Mar 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



216.239. hit all my main pages across 5 domains, but I'm getting one hit about every 3-4 hours, if at all, on dynamic pages. But it appears to slowly be getting every id=.

Maybe deepbots' new-and-improved ability to crawl more dynamic pages keeping them too busy to hang out as long?

leifwessman

3:49 pm on Mar 17, 2003 (gmt 0)

10+ Year Member




I mostly get 10 - 20 hits / min. Sometimes it falls to 1-3 hits / min and sometimes it's up to 30 / min. All my pages are dynamic.

Leif

Brett_Tabke

3:54 pm on Mar 17, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



We were just talking about how agressive GB had been this month. They hit here for 1900 pages an hour two nights ago. Not a problem when when it's 1-5am, but that is still up from the previous two big crawls.

uber_boy

4:02 pm on Mar 17, 2003 (gmt 0)

10+ Year Member



That's interesting, Brett. As I noted above, less than six months ago I'd have Google asking for 2-3000 pages/hour for two days straight. I still get brief bursts at this intensity, but now they last for maybe half an hour. The Googlebots then wait a few minutes and return for another session. The result is that the hourly rate rarely goes much beyond 1000. Fortunately, this cycle continues for several days.

For what it's worth, all my pages are dynamic but the URLs are massaged to look like static pages...

kwngian

5:05 pm on Mar 17, 2003 (gmt 0)

10+ Year Member




Sure beats being hit by CodeRed all day long.

kwngian

Jesse_Smith

6:18 pm on Mar 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Last Satuurday I got bombed by him, it took my server load up to 24.56, 10.49, 5.14.

plasma

6:26 pm on Mar 17, 2003 (gmt 0)

10+ Year Member



When googlebot is on our servers, it grabs a page about every 1.5 min. Same for dynamic and static content.