dstiles - 8:15 pm on Jul 5, 2011 (gmt 0)
I record all legit bot hits in a specific composite-site log (ie across the whole server in one log). I view this log several times a day. I would notice if the rate were more than two or three pages per second.
The same applies to the major bot companies that are using non-bot rDNS - I log all "bad" site hits including scrapers and server farms.
My experience is that the major bots tend not to hit the same IP at the same time: ie they scan one site, wait a while, then scan another site. The msnbot specifically scans a whole site at one sitting (I have no delay factors in most of my robots.txt files) then comes back a few minutes or hours later for another site.
NOTE: This only applies to web PAGES. It does not include images, css, js etc.