homepage Welcome to WebmasterWorld Guest from 54.166.128.254
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
msnbot gone crazy?
Nov 8th 2011 - at around 03:00 UTC
gethan

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4384688 posted 4:04 am on Nov 8, 2011 (gmt 0)

I'm currently getting hammered so badly by msnbot - that our site is barely staying up.

As a temporary solution I'm returning 503 to msnbot user-agents.

Anyone else seeing this?

 

gethan

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4384688 posted 4:07 am on Nov 8, 2011 (gmt 0)

More details: around 10 requests a second... like a DOS attack. I emailed abuse@msn.com - but don't expect much if anything...


www.example.org:80 207.46.194.123 - - [08/Nov/2011:04:04:54 +0000] "GET /Example/3458472 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.82 - - [08/Nov/2011:04:04:54 +0000] "GET /Example/2010978 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.79 - - [08/Nov/2011:04:04:54 +0000] "GET /Example/704586 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.125 - - [08/Nov/2011:04:04:54 +0000] "GET /Example/5822984 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.204.128 - - [08/Nov/2011:04:04:55 +0000] "GET /Example/546760 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.48 - - [08/Nov/2011:04:04:55 +0000] "GET /Example/5453835 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.195.167 - - [08/Nov/2011:04:04:55 +0000] "GET /Example/1667704 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.102 - - [08/Nov/2011:04:04:55 +0000] "GET /Example/3574573 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.67 - - [08/Nov/2011:04:04:55 +0000] "GET /Example/4814833 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
www.example.org:80 207.46.194.69 - - [08/Nov/2011:04:04:55 +0000] "GET /Example/1161610 HTTP/1.1" 503 20 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"

johnh123

5+ Year Member



 
Msg#: 4384688 posted 6:21 pm on Nov 11, 2011 (gmt 0)

i've been getting this on and off for the last week - any ideas as to what is going on or what can we do to stop it?

bingdude



 
Msg#: 4384688 posted 9:04 pm on Nov 11, 2011 (gmt 0)

Responded to both via IM. ;)

SEOPTI

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4384688 posted 6:55 am on Nov 13, 2011 (gmt 0)

gethan:

You are member since 2001, I believe you should have your own server and even under biggest crawler load you will stay at 1% CPU and 1% RAM usage. Or are you with 1000+ other people on a VPS machine which consts $ 3.99/month, if this is the case I believe website creation is your hobby.

I see no problem at all, Bing is hitting my sites with 100k+ bot requests/day.

You should feel lucky and thankful they try to crawl and index your site, send you traffic, send you conversions, people are crying because they don't see bingbot for months.

Sending mails to their abuse department because they crawl you hard is pointless and complete nonsense.

If you don't want their traffic just set your crawl rate in webmaster tools as low as possible or block them in robots.txt.

I love Bing traffic and their awesome traffic conversion levels so I go for the highest possible crawl rate day and night.

If you are completely unable to handle few bingbot requests how do you treat GBot and scraper requests, they are hitting your site 10 times harder.

Look at your code, probably your php/mysql code/queries were poorly written or you have not optimized your DB (indexes ...).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved