dstiles - 7:47 pm on Dec 29, 2011 (gmt 0)

I have always been under the impression that the default rate for all "reasonable" bots was a maximum rate of 2 pages/second. Depending on whether it grabs images etc and how many of those per page the max bot hit-rate for a site can obviously exceed that but the page rate should not.

That is not to say the crawl rate cannot be upped by crawl-rate directives if YOU wish.