Welcome to WebmasterWorld Guest from 3.80.4.76

Forum Moderators: Ocean10000

Message Too Old, No Replies

ntwk.msn.net

claims to be a BingBot

     
6:13 pm on Dec 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1992
votes: 74


So 157-55-39-132.search.msn.com.po18-218.co2-6nf-srch-2b.ntwk.msn.net , that is the RDNS I get.

Been hammering 2 of my sites for the past several days.

Is this a real BingBot or a knockoff from MSFT Ranges?

157.55.0.0/16 has several BingBot IPs with proper RDNS at this point.
6:26 pm on Dec 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts: 2661
votes: 103


Internal MSFT development? Does the UA differ from the normal Bingbot (mobile version of UA etc)?

Regards...jmcc
6:29 pm on Dec 12, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15812
votes: 848


Is this a real BingBot or a knockoff from MSFT Ranges?
Well, does it claim to be the bingbot?
7:56 pm on Dec 12, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


msnbot-157-55-39-132.search.msn.com

Valid crawl range for anything MSN

So yes, I would accept this as a valid Bingbot.
2:19 pm on Dec 13, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1992
votes: 74


This is the UA: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

if I do NSLOOKUP from where I am it resolves to msnbot-157-55-39-132.search.msn.com.

Problem is if I use it on my server via java.net.InetAddress.getByName('157.55.39.132') it resolves to 157-55-39-132.search.msn.com.po18-218.co2-6nf-srch-2b.ntwk.msn.net

There are tons of requests from surrounding /24 but only this IP is a problem and it seems like an intermittent issue.

I did some digging around and it seems the first time this IP was used to crawl this site was on 05/22/16. Since then it had accessed various URIs on the site successfully 96 times(resolved properly), including 2 dozen times in December of this year.

A head scratch-erererer for me here. :(
2:43 pm on Dec 13, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


Different look-ups will often report differently. At least java.net.InetAddress.getByName is getting it close.
6:13 pm on Dec 13, 2017 (gmt 0)

Junior Member

10+ Year Member

joined:June 25, 2005
posts:195
votes: 2


One IP address, two PTR records:

dig -x 157.55.39.132 @9.9.9.9
;; ANSWER SECTION:
132.39.55.157.in-addr.arpa. 3600 IN PTR po18-218.co2-6nf-srch-2b.ntwk.msn.net.
132.39.55.157.in-addr.arpa. 3600 IN PTR msnbot-157-55-39-132.search.msn.com.
6:44 pm on Dec 13, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15812
votes: 848


There are tons of requests from surrounding /24 but only this IP is a problem and it seems like an intermittent issue.
What's the nature of the problem? Is it showing un-bingbot-like behavior?

Cursory log search turns up tens of thousands of visits (small site) from 157.55.39 over the last few years. Never saw any reason to single them out. They seem to have started using this /24 pretty abruptly in late May 2014.

:: further, narrower search ::

Yup, I've met 157.55.39.132 (exactly). Only from the bingbot--both common and iPhone--not from other agents such as the plainclothes bingbot.
2:41 am on Dec 14, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1992
votes: 74


@thetrasher - that is right, and that is my point at this time- 2 records

The logic I use matches last n-chars to be = to .search.msn.com for this /16 range. If it is then the IP is OK to allow the content to be served to this IP. I dont do Reverse then Forward lookups - too expensive on CPU.

Sometimes it resolves to msnbot-157-55-39-132.search.msn.com, and lately it does not but not always. But only for this particular IP out of entire /24.

When it does not - the request gets 403.

There are tons of none BingBot/rouge allocated IPs in that /16 btw.
4:56 am on Dec 14, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15812
votes: 848


For bing I tend to block the other way around: If it's from a bing/msn range but doesn't call itself bingbot--or something similarly legitimate like site-verification--they risk getting locked out. Or at least redirected until such a time as they tell the world what they're up to (looking at you, 204.79).
3:36 pm on Dec 14, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1992
votes: 74


Thank Lucy, will have to look into it.

My rules are "trying" to stay the same for all bots that I allow:

From known Range > n-chars on the right of the string match what is accepted for RDNS >> UA matches a known bot >>> PASS. If UA does not match then it sets up an alert so I could see the difference in behavior, but still let them thru depending on a speed they try to access(abuse of crawl delay limit in robots.txt gets them a DROP REQUEST on IIS LEVEL per IP).

I was thinking there might be a hick-up/glitch in RDNS look-ups up the pipe for my hosting server.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members