tangor - 6:53 am on Dec 29, 2011 (gmt 0)

Most of my response was to your declaration of a `12 pages/sec' index being OK, and even apparently belittling the idea that to crawl at that rate may be abusive. I therefore thought that I should add a little more substantia to my claims, and at the same time highlight an on-coming issue that I've seen little commented-upon elsewhere.

Works for me. As I said, I allow Bing. If Bing romps at 12p/s and gets out of the way, that okay with me. After all, I let them in. So, It's all okay with me.

This solves the immediate problem (if there really is one):

Disallow: Bing

Or one can live with it. The log file listed above indicates you're returning a 403 which is about 500ish BYTES -- which can't get anywhere NEAR 818+GB. Sorry, this seems like a tempest in a teapot as regards Bing. Since Bing is the next best thing to Google what's the problem? Please, let us compare apples to apples.