Welcome to WebmasterWorld Guest from 54.159.165.175

Forum Moderators: mack

Stop DOS attack from MSN bot

   
3:52 am on Dec 8, 2012 (gmt 0)

10+ Year Member



Recently MSN bot keeps crawling my site almost all day long and sends the loading to the roof. It is really a DOS attack. I googled it and found many people have the same problem. How can I stop this from happening? Using robots.txt? I also heard the MSN bot does not follow robots.txt and heaves really bad.

BTW, whenever i see the word bing, I want to vomit.
3:46 am on Dec 15, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Did you determine it was MSNbot by the User Agent or IP?
11:22 pm on Dec 15, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Newbies - I suggest for a better over-view you read the Search Engine Spider and User Agent Identification forum - lots of recent stuff there about MSN/bing.

From my own viewpoint I see no more over-browsing than from googlebot.

If it's not a bot IP (check rDNS to see that) AND it's not a bot UA (there is only one I accept but there are two or three valid ones) then you should be rejecting access.
1:49 am on Dec 16, 2012 (gmt 0)

5+ Year Member



msnbot (and bingbot) will obey the crawl-delay directive from the robots.txt file. This will slow it down and prevent the DOS. It has worked wonders for me.
9:26 pm on Apr 20, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Did you determine it was MSNbot by the User Agent or IP?


I've been seeing the same thing lately and it's most certainly MSNbot and Bingbot from both IP addresses (do they really need so many?) and the UA. The also send a no-cache header which really messes up my squid front end.
4:36 pm on Sep 4, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We observe 1000+ parallel connections by msnbot. It's insane and unacceptable.

And yes, it really is msnbot based on ip
6:15 pm on Sep 4, 2013 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Free load testing, aren't they nice!
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month