Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: mack
Disallowing mnsbot in the robots.txt file has stopped the bandwidth usage.
I received a response from ms and they suggested setting a "crawl delay" of up to 600 seconds. Their bot ran up 355624 hits in less than 48 hours, more than 2 a second, on a site that only has 64 pages!
Now I understand that in the big picture this is only 4 gig of bandwidth and at $3 a gig only really cost me about $12. It's more of a pain in the butt than anything. This is a small site and I don't allocate much bandwidth to it, probably less than 4 gig a year, so to have msnbot suck all that up in 48 hours is certainly frustrating.
Maybe I should let the msnbot go and put the site up for sale: "Small site for sale, only 64 pages but over 5,000,000 hits this month!"
I wonder if I can train msnbot to click on adsense?