New MSN Mobile Crawler UA

Launched without fanfare

[Samizdata]

Coming from Microsoft on 65.55.4.nnn and presumably a new mobile crawler:

T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm)

Took robots.txt and headed in the right direction.

No information at the URL given (it redirects to Windows Live Help which doesn't work in Safari anyway) and no information anywhere else on the web as far as I can see.

A simple page explaining what it is would have been polite.

..

[GaryK]

Thanks for the heads-up.

ADDED: Have you got a list of these mobile bot UAs from MSN please?

[edited by: GaryK at 7:46 pm (utc) on Feb. 15, 2009]

[Samizdata]

Have you got a list of these mobile bot UAs from MSN please?

These are the only ones I know of:

Current standard:
MSMOBOT/1.1 (+http://search.msn.com/msnbot.htm)

New T-Mobile:
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm)

Discontinued?:
MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)

They seem to use two IP ranges, 65.55 (regular) and 202.96 (China).

...

[GaryK]

Thanks, that means I've got them all.

[GaryK]

Because I'm linking to this thread via TinyURL and Twitter I wanna be clear about the user agent, absent the smiley junk that WebmasterWorld adds to it.

T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x24;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm)

[GaryK]

Quick update to let you all know the format for this UA has changed slightly. A friend of mine let me know about it just a few minutes ago.

T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240; MSNBOT-MOBILE/1.1; +http://search.msn.com/msnbot.htm)

[GaryK]

Sorry for all the posts. I keep getting updated information about this user agent.

Sam, can you please confirm that the IP Address you posted was indeed the remote address and not the one in the X-FORWARDED-FOR header? Cause I'm being told now that a friend of mine is seeing that IP Addy in the headers, but that the remote address, 66.7.198.nn, is from a site belonging to a Brazilian company.

I still haven't seen this one yet so I can't say for sure.

Thanks. :)

[Samizdata]

Sorry, I don't usually log that data.

What I can say is that the logged IP is always Redmond, that robots.txt is always respected, and that only mobile-specific content (listed in a mobile sitemap) is fetched.

It quacks like a duck here.

...

[GaryK]

That sounds legit enough to me. The other one must be spoofed. Surprising it's being spoofed so soon.

[jdMorgan]

> The other one must be spoofed.

Which one is legitimate?
Which one is spoofed?

Jim

[GaryK]

The one I posted appears to be spoofed.

Apparently legit:
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm)

Apparently spoofed:
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240; MSNBOT-MOBILE/1.1; +http://search.msn.com/msnbot.htm)

The only thing that bothers me about the first one is the semi-colon after the screen dimensions. I wouldn't expect to see that in a legit user agent from Microsoft. But if Sam says he's got data that indicates it's from a MS IP Address space I have to believe him. Mine is most definitely from a Brazilian company being hosted by HostDime.com, Inc. I didn't realize that when I first posted it.

[edited by: GaryK at 1:47 am (utc) on Mar. 23, 2009]

[Pfui]

Legit UA seen today:

msnbot-65-55-51-20.msn.com
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240; MSNBOT-MOBILE/1.1; +http://search.msn.com/msnbot.htm)

robots.txt: YES