Welcome to WebmasterWorld Guest from 54.159.50.111

Forum Moderators: mack

Message Too Old, No Replies

Stop DOS attack from MSN bot

     
3:52 am on Dec 8, 2012 (gmt 0)

Full Member

10+ Year Member

joined:Apr 19, 2003
posts: 282
votes: 0


Recently MSN bot keeps crawling my site almost all day long and sends the loading to the roof. It is really a DOS attack. I googled it and found many people have the same problem. How can I stop this from happening? Using robots.txt? I also heard the MSN bot does not follow robots.txt and heaves really bad.

BTW, whenever i see the word bing, I want to vomit.
3:46 am on Dec 15, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Nov 11, 2007
posts:769
votes: 1


Did you determine it was MSNbot by the User Agent or IP?
11:22 pm on Dec 15, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3092
votes: 2


Newbies - I suggest for a better over-view you read the Search Engine Spider and User Agent Identification forum - lots of recent stuff there about MSN/bing.

From my own viewpoint I see no more over-browsing than from googlebot.

If it's not a bot IP (check rDNS to see that) AND it's not a bot UA (there is only one I accept but there are two or three valid ones) then you should be rejecting access.
1:49 am on Dec 16, 2012 (gmt 0)

Junior Member

10+ Year Member

joined:May 8, 2005
posts: 158
votes: 0


msnbot (and bingbot) will obey the crawl-delay directive from the robots.txt file. This will slow it down and prevent the DOS. It has worked wonders for me.
9:26 pm on Apr 20, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 20, 2004
posts:875
votes: 2


Did you determine it was MSNbot by the User Agent or IP?


I've been seeing the same thing lately and it's most certainly MSNbot and Bingbot from both IP addresses (do they really need so many?) and the UA. The also send a no-cache header which really messes up my squid front end.
4:36 pm on Sept 4, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 16, 2002
posts: 2010
votes: 0


We observe 1000+ parallel connections by msnbot. It's insane and unacceptable.

And yes, it really is msnbot based on ip
6:15 pm on Sept 4, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:June 20, 2006
posts:1878
votes: 5


Free load testing, aren't they nice!