g1smd - 12:44 am on Dec 29, 2011 (gmt 0)

Rather than measuring a nebulous pages per second rate without reference to size of pages or total number of pages on the site, nor for how long this rate was sustained and how often it was repeated, perhaps those and some other units of measure may be more useful, both by individual IP and by each searchengine?

How much bandwidth usage per searchengine per month is acceptable, how long between succesive pulls of the same page, and what percentage of the overall bandwidth is used by each search engine?

Quantify the access in these other terms then see if the picture looks better... or even worse.