homepage Welcome to WebmasterWorld Guest from 54.166.95.146
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
Stop DOS attack from MSN bot
newbies




msg:4526046
 3:52 am on Dec 8, 2012 (gmt 0)

Recently MSN bot keeps crawling my site almost all day long and sends the loading to the roof. It is really a DOS attack. I googled it and found many people have the same problem. How can I stop this from happening? Using robots.txt? I also heard the MSN bot does not follow robots.txt and heaves really bad.

BTW, whenever i see the word bing, I want to vomit.

 

ZydoSEO




msg:4528321
 3:46 am on Dec 15, 2012 (gmt 0)

Did you determine it was MSNbot by the User Agent or IP?

dstiles




msg:4528540
 11:22 pm on Dec 15, 2012 (gmt 0)

Newbies - I suggest for a better over-view you read the Search Engine Spider and User Agent Identification forum - lots of recent stuff there about MSN/bing.

From my own viewpoint I see no more over-browsing than from googlebot.

If it's not a bot IP (check rDNS to see that) AND it's not a bot UA (there is only one I accept but there are two or three valid ones) then you should be rejecting access.

jskrewson




msg:4528577
 1:49 am on Dec 16, 2012 (gmt 0)

msnbot (and bingbot) will obey the crawl-delay directive from the robots.txt file. This will slow it down and prevent the DOS. It has worked wonders for me.

eeek




msg:4566701
 9:26 pm on Apr 20, 2013 (gmt 0)

Did you determine it was MSNbot by the User Agent or IP?


I've been seeing the same thing lately and it's most certainly MSNbot and Bingbot from both IP addresses (do they really need so many?) and the UA. The also send a no-cache header which really messes up my squid front end.

amznVibe




msg:4607296
 4:36 pm on Sep 4, 2013 (gmt 0)

We observe 1000+ parallel connections by msnbot. It's insane and unacceptable.

And yes, it really is msnbot based on ip

RhinoFish




msg:4607312
 6:15 pm on Sep 4, 2013 (gmt 0)

Free load testing, aren't they nice!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved