msnbot is eating up a ton of my bandwidth

anyone else?

[limoshawn]

i keep getting bandwidth notices on one of my site so i looked into it and it turns out that i have 329809 hits from the msnbot for a total of 3.48GB of bandwidth this month. for this site Googlebot has only hit 158 times for 622KB, just for comparison.

any thoughts

[limoshawn]

I've disallowed msnbot in my robots.txt file, hopefully this will help. I did not previously have a robots.txt file and checking my error logs it looks like the bot was stuck in some kind of loop trying to check for the robots.txt over and over. I can't imagine what would cause this.

[phranque]

have you check to see what http response is returned when you HTTP GET robots.txt?

[simey]

I'd bet msn bot was prompting many to create a robots.txt or block ips.

[limoshawn]

phranque, before adding the robots.txt file the server return a 404 error.

Disallowing mnsbot in the robots.txt file has stopped the bandwidth usage.

I received a response from ms and they suggested setting a "crawl delay" of up to 600 seconds. Their bot ran up 355624 hits in less than 48 hours, more than 2 a second, on a site that only has 64 pages!
Now I understand that in the big picture this is only 4 gig of bandwidth and at $3 a gig only really cost me about $12. It's more of a pain in the butt than anything. This is a small site and I don't allocate much bandwidth to it, probably less than 4 gig a year, so to have msnbot suck all that up in 48 hours is certainly frustrating.

Maybe I should let the msnbot go and put the site up for sale: "Small site for sale, only 64 pages but over 5,000,000 hits this month!"
I wonder if I can train msnbot to click on adsense?