i keep getting bandwidth notices on one of my site so i looked into it and it turns out that i have 329809 hits from the msnbot for a total of 3.48GB of bandwidth this month. for this site Googlebot has only hit 158 times for 622KB, just for comparison.
I've disallowed msnbot in my robots.txt file, hopefully this will help. I did not previously have a robots.txt file and checking my error logs it looks like the bot was stuck in some kind of loop trying to check for the robots.txt over and over. I can't imagine what would cause this.
phranque, before adding the robots.txt file the server return a 404 error.
Disallowing mnsbot in the robots.txt file has stopped the bandwidth usage.
I received a response from ms and they suggested setting a "crawl delay" of up to 600 seconds. Their bot ran up 355624 hits in less than 48 hours, more than 2 a second, on a site that only has 64 pages! Now I understand that in the big picture this is only 4 gig of bandwidth and at $3 a gig only really cost me about $12. It's more of a pain in the butt than anything. This is a small site and I don't allocate much bandwidth to it, probably less than 4 gig a year, so to have msnbot suck all that up in 48 hours is certainly frustrating.
Maybe I should let the msnbot go and put the site up for sale: "Small site for sale, only 64 pages but over 5,000,000 hits this month!" I wonder if I can train msnbot to click on adsense?