homepage Welcome to WebmasterWorld Guest from 54.227.34.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
msnbot is eating up a ton of my bandwidth
anyone else?
limoshawn

5+ Year Member



 
Msg#: 3601763 posted 10:13 pm on Mar 15, 2008 (gmt 0)

i keep getting bandwidth notices on one of my site so i looked into it and it turns out that i have 329809 hits from the msnbot for a total of 3.48GB of bandwidth this month. for this site Googlebot has only hit 158 times for 622KB, just for comparison.

any thoughts

 

limoshawn

5+ Year Member



 
Msg#: 3601763 posted 2:10 am on Mar 16, 2008 (gmt 0)

I've disallowed msnbot in my robots.txt file, hopefully this will help. I did not previously have a robots.txt file and checking my error logs it looks like the bot was stuck in some kind of loop trying to check for the robots.txt over and over. I can't imagine what would cause this.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3601763 posted 5:24 am on Mar 16, 2008 (gmt 0)

have you check to see what http response is returned when you HTTP GET robots.txt?

simey

5+ Year Member



 
Msg#: 3601763 posted 8:21 am on Mar 16, 2008 (gmt 0)

I'd bet msn bot was prompting many to create a robots.txt or block ips.

limoshawn

5+ Year Member



 
Msg#: 3601763 posted 12:26 pm on Mar 16, 2008 (gmt 0)

phranque, before adding the robots.txt file the server return a 404 error.

Disallowing mnsbot in the robots.txt file has stopped the bandwidth usage.

I received a response from ms and they suggested setting a "crawl delay" of up to 600 seconds. Their bot ran up 355624 hits in less than 48 hours, more than 2 a second, on a site that only has 64 pages!
Now I understand that in the big picture this is only 4 gig of bandwidth and at $3 a gig only really cost me about $12. It's more of a pain in the butt than anything. This is a small site and I don't allocate much bandwidth to it, probably less than 4 gig a year, so to have msnbot suck all that up in 48 hours is certainly frustrating.

Maybe I should let the msnbot go and put the site up for sale: "Small site for sale, only 64 pages but over 5,000,000 hits this month!"
I wonder if I can train msnbot to click on adsense?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved