Msg#: 3601763 posted 10:13 pm on Mar 15, 2008 (gmt 0)
i keep getting bandwidth notices on one of my site so i looked into it and it turns out that i have 329809 hits from the msnbot for a total of 3.48GB of bandwidth this month. for this site Googlebot has only hit 158 times for 622KB, just for comparison.
Msg#: 3601763 posted 2:10 am on Mar 16, 2008 (gmt 0)
I've disallowed msnbot in my robots.txt file, hopefully this will help. I did not previously have a robots.txt file and checking my error logs it looks like the bot was stuck in some kind of loop trying to check for the robots.txt over and over. I can't imagine what would cause this.
Msg#: 3601763 posted 12:26 pm on Mar 16, 2008 (gmt 0)
phranque, before adding the robots.txt file the server return a 404 error.
Disallowing mnsbot in the robots.txt file has stopped the bandwidth usage.
I received a response from ms and they suggested setting a "crawl delay" of up to 600 seconds. Their bot ran up 355624 hits in less than 48 hours, more than 2 a second, on a site that only has 64 pages! Now I understand that in the big picture this is only 4 gig of bandwidth and at $3 a gig only really cost me about $12. It's more of a pain in the butt than anything. This is a small site and I don't allocate much bandwidth to it, probably less than 4 gig a year, so to have msnbot suck all that up in 48 hours is certainly frustrating.
Maybe I should let the msnbot go and put the site up for sale: "Small site for sale, only 64 pages but over 5,000,000 hits this month!" I wonder if I can train msnbot to click on adsense?