tangor - 5:23 am on Dec 29, 2011 (gmt 0)
tangor, I do not think that you have thought this through. I've been stopping abuse from bots for several years now, and reporting it for 18 months. Here are some recent facts from the last year for you to consider:
Granted. I got serious about this sometime around 2000. That said, Your HURT is apparently different than mine. I certainly don't have 130k plus pages to be sucked (see above). But then again, the Bing suck only happens on a routine basis and all my pages are indexed in short order and is done. Different strokes, and no heart strokes for me. As always YMMV.
My comment was specific: Be indexed or NOT... your choice by throttle, .htaccess, or Crawl-Delay. No skin off my nose. :)
Let's not make this a Your Hurt is bigger than Mine. I promise my "hurt" is way less! I only allow FIVE (5) bots (white list). Period. Thus all those others mentioned never bothered me. (I have thought this out) Anything else is nuked with prejudice. I don't make any attempt to coddle any bots other than my select few in hopes of possible "human" traffic most likely from countries which have no valuation (conversions). Hence my Monthly Bandwidth is well within control, etc. etc. etc.
You pick and choose your battles. I start with Best Defense First (what I allow), then relax IF THERE IS A REASON TO DO SO. That has happened only once... and lasted a year then was shut down.
Bing is not a problem FOR ME... that's where my traffic is coming from these days. Again, pick and choose... but I do feel your pain. What to block? What to block? What to...