ntwk.msn.net

claims to be a BingBot

[blend27]

So 157-55-39-132.search.msn.com.po18-218.co2-6nf-srch-2b.ntwk.msn.net , that is the RDNS I get.

Been hammering 2 of my sites for the past several days.

Is this a real BingBot or a knockoff from MSFT Ranges?

157.55.0.0/16 has several BingBot IPs with proper RDNS at this point.

[jmccormac]

Internal MSFT development? Does the UA differ from the normal Bingbot (mobile version of UA etc)?

Regards...jmcc

[lucy24]

Is this a real BingBot or a knockoff from MSFT Ranges?

Well, does it claim to be the bingbot?

[keyplyr]

msnbot-157-55-39-132.search.msn.com

Valid crawl range for anything MSN

So yes, I would accept this as a valid Bingbot.

[blend27]

This is the UA: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

if I do NSLOOKUP from where I am it resolves to msnbot-157-55-39-132.search.msn.com.

Problem is if I use it on my server via java.net.InetAddress.getByName('157.55.39.132') it resolves to 157-55-39-132.search.msn.com.po18-218.co2-6nf-srch-2b.ntwk.msn.net

There are tons of requests from surrounding /24 but only this IP is a problem and it seems like an intermittent issue.

I did some digging around and it seems the first time this IP was used to crawl this site was on 05/22/16. Since then it had accessed various URIs on the site successfully 96 times(resolved properly), including 2 dozen times in December of this year.

A head scratch-erererer for me here. :(

[keyplyr]

Different look-ups will often report differently. At least java.net.InetAddress.getByName is getting it close.

[thetrasher]

One IP address, two PTR records:

dig -x 157.55.39.132 @9.9.9.9

;; ANSWER SECTION:
132.39.55.157.in-addr.arpa. 3600 IN  PTR  po18-218.co2-6nf-srch-2b.ntwk.msn.net.
132.39.55.157.in-addr.arpa. 3600 IN  PTR  msnbot-157-55-39-132.search.msn.com.

[lucy24]

There are tons of requests from surrounding /24 but only this IP is a problem and it seems like an intermittent issue.

What's the nature of the problem? Is it showing un-bingbot-like behavior?

Cursory log search turns up tens of thousands of visits (small site) from 157.55.39 over the last few years. Never saw any reason to single them out. They seem to have started using this /24 pretty abruptly in late May 2014.

:: further, narrower search ::

Yup, I've met 157.55.39.132 (exactly). Only from the bingbot--both common and iPhone--not from other agents such as the plainclothes bingbot.

[blend27]

@thetrasher - that is right, and that is my point at this time- 2 records

The logic I use matches last n-chars to be = to .search.msn.com for this /16 range. If it is then the IP is OK to allow the content to be served to this IP. I dont do Reverse then Forward lookups - too expensive on CPU.

Sometimes it resolves to msnbot-157-55-39-132.search.msn.com, and lately it does not but not always. But only for this particular IP out of entire /24.

When it does not - the request gets 403.

There are tons of none BingBot/rouge allocated IPs in that /16 btw.

[lucy24]

For bing I tend to block the other way around: If it's from a bing/msn range but doesn't call itself bingbot--or something similarly legitimate like site-verification--they risk getting locked out. Or at least redirected until such a time as they tell the world what they're up to (looking at you, 204.79).

[blend27]

Thank Lucy, will have to look into it.

My rules are "trying" to stay the same for all bots that I allow:

From known Range > n-chars on the right of the string match what is accepted for RDNS >> UA matches a known bot >>> PASS. If UA does not match then it sets up an alert so I could see the difference in behavior, but still let them thru depending on a speed they try to access(abuse of crawl delay limit in robots.txt gets them a DROP REQUEST on IIS LEVEL per IP).

I was thinking there might be a hick-up/glitch in RDNS look-ups up the pipe for my hosting server.