BingBot Activities

[Dimitri]

Hi,

Though to chat a little bit about the little Bing guy, since no one seems to mind much about him :)

Since mid July, I see a huge BingBot activity at my site x10 compared to the "normal".

A very useful information is, that I have no idea why.

By the way, my concerns about BingBot, is that, half of requests are coming from IPs with no reverse DNS . I use FCrDNS to verify that a robot is legitimate (and not faked UA), so these requests are automatically blocked by default, until I whitelist them manually. Also it's good practice to have reverse DNS for all IPs , I wonder why Microsoft is not bothering doing it ...

So between Facebook pretending to be Bing, and Bing not event using an IP which could easily be spotted as being legitimate, this is odd behaviors .

Is there a list of official BingBot IP ranges?

[not2easy]

The only list I have ever met is admittedly very old and limited: [webmasterworld.com...] which lists

MSN has blocks
64.4.0.0 - 64.4.63.255
207.68.128.0 - 207.68.207.255
65.52.0.0 - 65.55.255.255
207.46.0.0 - 207.46.255.255

I do not know why MS does not wish to define where their bots should come from and instead seems to think that UA should do the trick. I have not looked into it for some time because of other anomalies. I have seen way too much activity from those ranges that I do not approve of to give their IPs carte blanche. Remember that the Bing's bot does not necessarily follow all those bot directives you may have added to your sites such as "nofollow", those are Google's idea.

As for ways to deal with their bots and whitelisting ideas, you can find starting points in the Search Engine Spider and User Agent Identification Forum - example: [webmasterworld.com...]

and the Apache Forum - example: [webmasterworld.com...]

There are many old scripting techniques in the PHP forum as well since bot control has been a webmaster thing since forever. They may require some gentle updating, but for ideas, here's one oldie: [webmasterworld.com...]

This probably isn't what you had hoped for, but it's a start.

[lucy24]

You forgot 157.55.whatever-it-is. Also 40.77.thingummy and 13.66.whatsit. Poor Microsoft* came in just too late to snabble one of those /8 blocks that many major tech firms have had from the outset, like Apple’s 17, so they have to grab micro-ranges where they can. Oddly, they don't seem to be using IPv6 addresses at all (unlike, say, Seznam, which is based in Europe and therefore understandably uses IPv6 wherever possible).

Lately--the past year or so--I’ve met a lot of bingbot activity from 52.162 and 52.240; these are either new acquisitions or existing ranges that have been repurposed for crawling. 23.whatsit used to be popular but is now rarely seen.

:: detour to log-wrangling scripts for exact numbers ::

I ignore bingbot from all of these:
13.66, 13.77; 23.96/13; 40.64/10; 65.52/14; 131.253; 157.54-60; 191.232.136; 199.30.16.0/20; 207.46

Why they need such a vast array of IPs must remain a mystery. To this day, Google does all its crawling from a single /20.


* A pair of words you do not see every day.

[iamlost]

1. just because it says bingbot and comes from a MSFT range is meaningless in these cloud days.

2. Verifying Bingbot [bing.com]

If you see what appears to be Bingbot traffic in your server logs based on a user agent string, for example Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm), and you want to know if this traffic really is originating from a Bing server, you can take the following steps:

1. Perform a reverse DNS lookup using the IP address from the logs to verify that it resolves to a name that end with search.msn.com
2. Do a forward DNS lookup using the name from step 1 to confirm that it resolves back to the same IP address

This is the official method of verification and can be done in real time.

Keeps things simple. :)

And yes, bingbot does do the 'crazy crawl' more often than not... all that Seattle coffee...

[tangor]

I just which Bing would take a nap from time to time.

Hitting me (and verified they are the culprit!) about 5 times more often than Auntie G....

Such Aggression! :)