Beyond PR, what else determines # of pages read?

[uber_boy]

This is my first post here -- I've been lurking for a while -- and I've searched the archives for an answer before asking this question but, even so, I suspect a helpful member will simply point me to a relevant post from times gone by. So with that said, as the subject line suggests, I'm interested in knowing if anything other than PR affects the number of pages googlebot examines.

I ask the foregoing questions because I operate a bibliographic site with millions of unique pages and, of course, would love to have each and every one of them indexed by google. The site's been a PR7 at least as long as I've known about PR -- about six months -- but the number of pages varies quite a bit from month to month. And whereas the site used to be subject to a couple of days of mind-numbing attention from googlebot, it now gets three or four days of less intense attention. Thus, I'm beginning to wonder whether googlebot's algorithm involves reading in as many pages as possible during a window of time determined by PR.

If what I'm proposing is crazy, please let me know as I'm contemplating a major hardware upgrade to accommodate googlebot. That said, let me finish this off by saying how incredibly helpful this forum has been for me!

[vitaplease]

uber_boy, welcome to WebmasterWorld

static links and high PR with well spread internal linking seem most important:

[webmasterworld.com...]

I would say a lot of deep linking from external sites to different internal pages within your site would help more than all the external inbound links going to your index page.

[uber_boy]

Thanks for the reply vitaplease and, in particular, the link to that earlier thread. However, I'm still in the dark about why the number of pages googlebot reads and the rate at which it reads them varies from month to month even though I'm making few if any changes to the site.

I'm interested because the last couple of months in particular, I've encountered a "kinder, gentler" googlebot. It still read about the same number of pages as its "more vicious" predecessor (i.e., about 125,000), but took about twice as long to do so. In response, you might say, who cares so long as the same number are being read? And in return, I might agree were it not for the fact that, until now, the number of pages read was increasing each month. Thus, before I go out and spend money on new hardware to serve up pages to googlebot more quickly, I guess I first wanted to check out whether the googlebot algorithm has changed and the old, more vicious googlebot has been replaced?

THanks in advance for any replies!

[ciml]

I haven't seen this but some people have mentioned that Google spiders more if the server responds quickly.

vitaplease's comments are, as we would expect, right on the button. Google change how they spider from month to month, so a change in the spidering pattern of your site may be due to them, not you.

[WebGuerrilla]

>>Google spiders more if the server responds quickly.

I think this is probably one of the most overlooked factors. Especially if you have a large site.

The faster you can serve pages to the bot, the more pages the bot can collect in the amount of time alloted for a given site.

I worked on a project recently where we moved a large site that was on a average box to a new box running dual xeons. The number of pages Google indexed doubled.

[uber_boy]

Thanks a bunch for that, WebGuerrilla! GIven the amount of money I'm going to have to spend, your words are VERY reassuring...