ZydoSEO
msg:4528321 | 3:46 am on Dec 15, 2012 (gmt 0) |
Did you determine it was MSNbot by the User Agent or IP?
|
dstiles
msg:4528540 | 11:22 pm on Dec 15, 2012 (gmt 0) |
Newbies - I suggest for a better over-view you read the Search Engine Spider and User Agent Identification forum - lots of recent stuff there about MSN/bing.
From my own viewpoint I see no more over-browsing than from googlebot.
If it's not a bot IP (check rDNS to see that) AND it's not a bot UA (there is only one I accept but there are two or three valid ones) then you should be rejecting access.
|
jskrewson
msg:4528577 | 1:49 am on Dec 16, 2012 (gmt 0) |
msnbot (and bingbot) will obey the crawl-delay directive from the robots.txt file. This will slow it down and prevent the DOS. It has worked wonders for me.
|
eeek
msg:4566701 | 9:26 pm on Apr 20, 2013 (gmt 0) |
| Did you determine it was MSNbot by the User Agent or IP? |
|
I've been seeing the same thing lately and it's most certainly MSNbot and Bingbot from both IP addresses (do they really need so many?) and the UA. The also send a no-cache header which really messes up my squid front end.
|
|
