MSN/Live MSMOBOT User-agent string change

User-agent string changed

[jdMorgan]

First seen today:

65.55.51.20 - - [10/Sep/2008:05:29:05 -0400] "GET /robots.txt HTTP/1.1" 200 1776 "-" "MSMOBOT/1.1 (+http://search.msn.com/msnbot.htm)"

The old user-agent string was:

MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)/1.1 (+http://search.msn.com/msnbot.htm)"

Nice, consistent version upgrade there... :(

Jim

[Samizdata]

I already see this from Microsft (China) Co.Ltd (note spelling) on 202.96.51.nnn - naughty.

Also spotted from 219.142.53.nn and likewise blocked.

...

[jdMorgan]

202.96.51.128-255 is Microsoft, spelling errors notwithstanding.
219.142.53.0-127 has no rDNS, but I've got it allowed as MS-China as well.

I don't want to be blocking their mobile crawlers because my site does support mobile devices.

My sole purpose in posting above was to note that the user-agent string had been updated.

Jim

[Samizdata]

Thanks for the education Jiim, and apologies if I misled anyone.

I have long supported mobile devices and have nothing against the people of China, but I
will be leaving the blocks in place - though my reasons will probably not apply to anyone else.

A couple of years ago the mobile section of one of my sites took off in a big way, so much so that the Nokia browser overtook Internet Explorer as the most frequent visitor (with Motorola and SonyEricsson not far behind).

I offer free small-screen videos in various formats (mp4, 3gp, wmv and realmedia) to any mobile device that can handle them, and demand was so high that I had to use a content distribution network to avoid incurring extra bandwidth charges.

Much of the traffic came from Asia, where use of the mobile web was (and remains) far in advance of western countries. I suddenly found myself hugely popular in China, India and Indonesia, as well as Russia and various countries in Africa, South America and the Middle East.

Unfortunately I was popular for the wrong reason.

To this day the site gets hammered by thumb-twiddlers from all over the globe who are under the misapprehension that my amusing little videos are "adult" content - which they absolutely are not, though this did not stop some extremely unsalubrious sites from linking to them.

So while I wouldn't normally block reputable crawlers, and enjoy getting visits from all parts of the planet, I will be better off if Microsft (sic) does not advertise my mobile content in the Orient.

I suspect that most such visitors will find my videos an "anti-climax" anyway.

...

[GaryK]

Jim, it's a minor change, but it seems they've changed it again to use upper-case even though the version number is the same:

MSMOBOT/1.1 (+http://search.msn.com/msnbot.htm)
65.55.51.15
msnbot-65-55-51-15.msn.com