Welcome to WebmasterWorld Guest from 54.197.94.141

Forum Moderators: mack

Message Too Old, No Replies

msnbot is eating up a ton of my bandwidth

anyone else?

   
10:13 pm on Mar 15, 2008 (gmt 0)

5+ Year Member



i keep getting bandwidth notices on one of my site so i looked into it and it turns out that i have 329809 hits from the msnbot for a total of 3.48GB of bandwidth this month. for this site Googlebot has only hit 158 times for 622KB, just for comparison.

any thoughts

2:10 am on Mar 16, 2008 (gmt 0)

5+ Year Member



I've disallowed msnbot in my robots.txt file, hopefully this will help. I did not previously have a robots.txt file and checking my error logs it looks like the bot was stuck in some kind of loop trying to check for the robots.txt over and over. I can't imagine what would cause this.
5:24 am on Mar 16, 2008 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



have you check to see what http response is returned when you HTTP GET robots.txt?
8:21 am on Mar 16, 2008 (gmt 0)

5+ Year Member



I'd bet msn bot was prompting many to create a robots.txt or block ips.
12:26 pm on Mar 16, 2008 (gmt 0)

5+ Year Member



phranque, before adding the robots.txt file the server return a 404 error.

Disallowing mnsbot in the robots.txt file has stopped the bandwidth usage.

I received a response from ms and they suggested setting a "crawl delay" of up to 600 seconds. Their bot ran up 355624 hits in less than 48 hours, more than 2 a second, on a site that only has 64 pages!
Now I understand that in the big picture this is only 4 gig of bandwidth and at $3 a gig only really cost me about $12. It's more of a pain in the butt than anything. This is a small site and I don't allocate much bandwidth to it, probably less than 4 gig a year, so to have msnbot suck all that up in 48 hours is certainly frustrating.

Maybe I should let the msnbot go and put the site up for sale: "Small site for sale, only 64 pages but over 5,000,000 hits this month!"
I wonder if I can train msnbot to click on adsense?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month