jdMorgan - 11:39 pm on Feb 25, 2010 (gmt 0)
If no other cause is found, and crawl-delay in robots.txt plus the above cache-, expiry-, and E-tag-header suggestions don't help and and this problem persists, you have the option to serve a 503-Service Unavailable response accompanied with a Retry-After header. If you set the Retry-After time at 5 to 15 seconds, your problem should be alleviated.
The above is based on the HTTP protocol. I personally have no idea whether msnbot will handle it correctly.
Also, don't complain too loudly to them about msnbot's behavior. I did that several years ago, and the site I was complaining about is still "banned" at Bing, although none of the techs can see any problem in the tools available to them, and all report that the site is *not* blocked, despite the fact that it no longer shows even for it's own domain, and there is a "Some results have been removed" message at the bottom of the screen. It's a non-profit, informational site, so I'm not losing any money (or sleep) over it.