Is This an Authentic Bingbot?

now crawling from 23

[lucy24]

This is not a new Bing/MSN range, but I've only recently started meeting the bingbot from here so I thought I'd mention it:

23.96.0.0/13
23.96-103

So far, every visit has been from the exact IP
23.103.64.39
so I don't know how much of the /13 is actually being used for crawling.

[keyplyr]

Probably someone using this range to fake Bingbot. This is not an official crawl range. None of this /13 is.

A lot of this is Azure, like AWS.

Do the headers match Bingbot from official ranges?

[lucy24]

If it's a faker, they have gone to a lot of trouble to send the identical headers in the identical order. There's no From: header, but this is not always present in any case.

Heaven knows there is no shortage of non-crawlers from MSN properties: witness the plainclothes bingbot and its pals from Drake Holdings.

[keyplyr]

...plainclothes bingbot and its pals from Drake Holdings

Personally, I don't consider those valid Bingbot... the bot that is purposed for indexing pages in the Bing Search index.

They may have some relation to MSN or even Bing itself, but those in that category should be judged on a case by case basis, like anything else.

I'll keep a look out for it.

[lucy24]

Personally, I don't consider those valid Bingbot

Well I did say “non-crawler”. The Drake entity annoys me to no end; I redirect it. Maybe in another five years they will get the message. Or maybe in another five years we'll get a fresh Bingdude and he'll explain what the bally thing wants.

[keyplyr]

Yes, I saw "non-crawler" but didn't understand your meaning. I guess I do now... I think.

[TravisDGarrett]

Bingbot, like other "serious" bots, can be verified by performing a reverse DNS lookup and for those paranoiac like me, then doing a forward DNS lookup.

In the case of Bingbot the reverse DNS has to resolve to xxx.search.msn.com
Source: [bing.com...]

Anything with reverse DNS lookup is fake bot.

[keyplyr]

Yes, TravisDGarrett that's what we're discussing... whether this Bingbot is authentic or not coming from a non-crawl range.

Sometimes authentic bots do come from non-assigned crawl ranges. Sometimes the ranges have recently been assigned as crawl but the data has yet to propagate to all the lookup services.

[wilderness]

FWIW, the range has been allocated to MSFT since at least early-2014

23.99.16.188 - - [16/Feb/2014:02:54:56 -0700] "GET / HTTP/1.1" 403 794 "-" "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 ( .NET4.0E)"
23.99.16.188 - - [16/Feb/2014:02:54:57 -0700] "GET / HTTP/1.1" 200 6028 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36"
23.99.16.188 - - [16/Feb/2014:02:54:57 -0700] "GET / HTTP/1.1" 200 6028 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36"

NOTHING is unusual for MS tactics going back to the 2003 threads in this same forum, and before Bing even existed.

[keyplyr]

FWIW, the range has been allocated to MSFT since at least early-2014

No one's debating that. However in that range is lots of IPs that are leased to whoever wants to use them (think AWS.)

[wilderness]

"(think AWS.)"

Not sure why you would assume otherwise.
The Class B's in that A offer more focus (may even be more) .
^23\.(8[0-389]|9[4-9]|10[0-9]|110)\.

[keyplyr]

I don't assume otherwise. Not sure why you would think so :)

[blend27]

@lucy24

I got the same IP trying to access some pages on one site. RDNS for 23.103.64.39 does not translate to a usual msnbot-NN-NNN-NN-NN.search.msn.com, so no Cheese for this bot for now. Oh, and this is the only IP from that /13. Started crawling on 03/04/18.

Also couple of hits from 40.77.169.22 with Bing UA.

[lucy24]

Started crawling on 03/04/18.

Yes, I think that was the first day I saw them too.

40.77 blahblah is also Bing. Just think, if Microsoft had been a teeny bit quicker off the mark back in 1990-whatever when IPv4 was first apportioned, they too could have got a whole /8 to themselves. Lots of companies (40 started out as Eli Lilly) and governmental units are currently selling theirs off. Instead they're stuck with little MSN slivers all over the map.

[keyplyr]

40.77 blahblah is also Bing

No, it is Microsoft. Same story here.

[lucy24]

Oops, yeah, I tend to equate them.

[keyplyr]

Giving this further thought... it is entirely possible Microsoft has become lazy (or for some other reason) and not officially allocating ranges as crawl.bingbot; just using any range owned by Microsoft to send out bingot.

I've just set up a series of tests to see if data crawled from a couple M$ ranges, but not crawl.bingbot, make it into Bing Webmaster Tools.

[keyplyr]

Still watching 23.96.0.0/13 for return of bingbot.

Seen only once on my sites on 20/Mar/2018