Forum Moderators: open

Message Too Old, No Replies

Is This an Authentic Bingbot?

now crawling from 23

         

lucy24

9:53 pm on Mar 11, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



This is not a new Bing/MSN range, but I've only recently started meeting the bingbot from here so I thought I'd mention it:

23.96.0.0/13
23.96-103

So far, every visit has been from the exact IP
23.103.64.39
so I don't know how much of the /13 is actually being used for crawling.

keyplyr

3:06 am on Mar 12, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Probably someone using this range to fake Bingbot. This is not an official crawl range. None of this /13 is.

A lot of this is Azure, like AWS.

Do the headers match Bingbot from official ranges?

lucy24

4:08 am on Mar 12, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If it's a faker, they have gone to a lot of trouble to send the identical headers in the identical order. There's no From: header, but this is not always present in any case.

Heaven knows there is no shortage of non-crawlers from MSN properties: witness the plainclothes bingbot and its pals from Drake Holdings.

keyplyr

4:51 am on Mar 12, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



...plainclothes bingbot and its pals from Drake Holdings
Personally, I don't consider those valid Bingbot... the bot that is purposed for indexing pages in the Bing Search index.

They may have some relation to MSN or even Bing itself, but those in that category should be judged on a case by case basis, like anything else.

I'll keep a look out for it.

lucy24

7:51 am on Mar 12, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Personally, I don't consider those valid Bingbot
Well I did say “non-crawler”. The Drake entity annoys me to no end; I redirect it. Maybe in another five years they will get the message. Or maybe in another five years we'll get a fresh Bingdude and he'll explain what the bally thing wants.

keyplyr

8:04 am on Mar 12, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yes, I saw "non-crawler" but didn't understand your meaning. I guess I do now... I think.

TravisDGarrett

12:30 pm on Mar 12, 2018 (gmt 0)



Bingbot, like other "serious" bots, can be verified by performing a reverse DNS lookup and for those paranoiac like me, then doing a forward DNS lookup.

In the case of Bingbot the reverse DNS has to resolve to xxx.search.msn.com
Source: [bing.com...]

Anything with reverse DNS lookup is fake bot.

keyplyr

9:04 pm on Mar 12, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yes, TravisDGarrett that's what we're discussing... whether this Bingbot is authentic or not coming from a non-crawl range.

Sometimes authentic bots do come from non-assigned crawl ranges. Sometimes the ranges have recently been assigned as crawl but the data has yet to propagate to all the lookup services.

wilderness

12:19 am on Mar 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



FWIW, the range has been allocated to MSFT since at least early-2014

23.99.16.188 - - [16/Feb/2014:02:54:56 -0700] "GET / HTTP/1.1" 403 794 "-" "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 ( .NET4.0E)"
23.99.16.188 - - [16/Feb/2014:02:54:57 -0700] "GET / HTTP/1.1" 200 6028 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36"
23.99.16.188 - - [16/Feb/2014:02:54:57 -0700] "GET / HTTP/1.1" 200 6028 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36"

NOTHING is unusual for MS tactics going back to the 2003 threads in this same forum, and before Bing even existed.

keyplyr

12:27 am on Mar 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



FWIW, the range has been allocated to MSFT since at least early-2014
No one's debating that. However in that range is lots of IPs that are leased to whoever wants to use them (think AWS.)

wilderness

12:20 pm on Mar 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



"(think AWS.)"

Not sure why you would assume otherwise.
The Class B's in that A offer more focus (may even be more) .
^23\.(8[0-389]|9[4-9]|10[0-9]|110)\.

keyplyr

12:25 pm on Mar 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I don't assume otherwise. Not sure why you would think so :)

blend27

1:20 pm on Mar 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



@lucy24

I got the same IP trying to access some pages on one site. RDNS for 23.103.64.39 does not translate to a usual msnbot-NN-NNN-NN-NN.search.msn.com, so no Cheese for this bot for now. Oh, and this is the only IP from that /13. Started crawling on 03/04/18.

Also couple of hits from 40.77.169.22 with Bing UA.

lucy24

8:32 pm on Mar 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Started crawling on 03/04/18.
Yes, I think that was the first day I saw them too.

40.77 blahblah is also Bing. Just think, if Microsoft had been a teeny bit quicker off the mark back in 1990-whatever when IPv4 was first apportioned, they too could have got a whole /8 to themselves. Lots of companies (40 started out as Eli Lilly) and governmental units are currently selling theirs off. Instead they're stuck with little MSN slivers all over the map.

keyplyr

10:06 am on Mar 15, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



40.77 blahblah is also Bing
No, it is Microsoft. Same story here.

lucy24

10:15 am on Mar 15, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Oops, yeah, I tend to equate them.

keyplyr

4:30 am on Mar 22, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Giving this further thought... it is entirely possible Microsoft has become lazy (or for some other reason) and not officially allocating ranges as crawl.bingbot; just using any range owned by Microsoft to send out bingot.

I've just set up a series of tests to see if data crawled from a couple M$ ranges, but not crawl.bingbot, make it into Bing Webmaster Tools.

keyplyr

7:45 pm on Mar 25, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Still watching 23.96.0.0/13 for return of bingbot.

Seen only once on my sites on 20/Mar/2018