the plainclothes bingbot

[lucy24]

#1 Quick heads-up for anyone who has been identifying the plainclothes bingbot in something like this form:

RewriteCond %{REMOTE_ADDR} ^(65\.5[2-5]|131\.253\.[2-4]\d|157\.(5[4-9]|60)|207\.46)\.
RewriteCond %{HTTP_USER_AGENT} MSIE\ \d\.0;\ Windows\ NT

Redmond has finally (!) caved in and allowed their robot to use MSIE 10. So from here on it's

MSIE\ \d\d?\.0

#2 In the course of looking into this, I discovered that this mysterybot* seems to have some connection with social media. (Which one belongs to Microsoft? I forget :() This is by no means its only job-- but each time I get a flurry of preliminary visits from places like Twitter and facebook, the plainclothes bingbot is right there too.


* Every time someone thinks they've figured out what it does, it pulls some entirely new trick out of its hat, wiping out the latest hypothesis.

[wilderness]

Hey lucy,
How goes it?

is that the complete UA?

RewriteCond %{HTTP_USER_AGENT} MSIE\ \d\.0;\ Windows\ NT

I've had an "ends with NT" and a couple of other "ends with" in place for a long while.

[lucy24]

No, it's just enough to put in a RewriteRule: "Comes from a Microsoft IP but appears to be a regular browser". No anchors.

:: shuffling papers ::

This latest one was, in full,

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)

(MSIE 10 is the first one to admit to Mozilla 5, isn't it?)

:: further shuffling ::

A few more random UAs (the _ means double space, a characteristic of the group):

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; _ SLCC1; _ .NET CLR 1.1.4322; _ .NET CLR 2.0.50727; _ .NET CLR 3.0.04506.648)

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; _ SV1; _ .NET CLR 1.1.4322; _ .NET CLR 2.0.40607; _ .NET CLR 3.0.30729; _ .NET CLR 3.5.30729; _ MS-RTC LM 8)

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; _ SLCC1; _ .NET CLR 1.1.4325; _ .NET CLR 2.0.50727; _ .NET CLR 3.0.30729; _ .NET CLR 3.5.30729; _ InfoPath.2)

et cetera. They're pretty humanoid. There's also an MSIE 8, but it only seems to work BingSiteAuth.xml so it's exempt from RewriteRules. I don't think I've met an MSIE 9.

[wilderness]

Seems to me that I've a Rewrite in place which prevents this kind of crap (Plain Jane's) from MS bot ranges.