tangor - 12:14 am on Dec 29, 2011 (gmt 0)
Whew! There's a fine line between having a site indexed, or crying abuse because that index is too "abusive". Not sure where that number is... 12 pages/sec... then done, or 12 days with no index... :(
Most of my sites are in the 2k page and under, so this is not a problem for me. I can see how it might affect significantly larger sites.
I do echo agreement that Crawl-Delay should be routinely supported without any requirement to sign up. Let's not fix something that shouldn't be broken (if properly implemented).