ms research robot

MSN strikes again. Again.

[Pfui]

207.46.92.19
ms research robot

robots.txt? NO

That address? Yep -- Microsoft Corp.

FWIW...

On the heels of the still jaw-droppingly misbehaved "msnbot/2.0b" which never met a robots.txt it bothered to heed, AND the stultifyingly tedious request process to get MSN to remove robots.txt-denied files and directories they improperly indexed (& then the requests to MSN only seem to stick for maybe a month or so, tops; then you get to go through the process all over again; and again), AND now at least two new bots in the past 10 days, IP-cloaked and featuring a 50% robots.txt-read failure rate --

I am *this* close to denying everything MSN bot-related and sending all MSN referers to the front door and all MSN bots to robots.txt/Disallow or the bit bucket.

Has anyone else put the kibosh on any/all of MSN's bots?

[jdMorgan]

I did, by complaining too much about pages disallowed in robots.txt being fetched, and pages with the meta-robots-noindex tag appearing in SERPs. As a courtesy, they completely removed my site(s) from the index. The msnbots still spider, but most pages where the sites should rank include the "Some results have been removed" message and no sign of my pages.

I reported the problems about a year ago, and the tech support ticket is still open and unresolved. In a few weeks, I'm going to block all thir 'bots because they're consuming lots of bandwidth for absolutely no return, and the sites rank quite well in the other major engines. This is an authoritative site and its mobile-device-version "twin" -- non-profit, no advertising, and only one cookie that allows new/unknown high-capability mobile devices to override the default mobile device redirection from the HTML site to the XML/XHTML-MP site.

Anyway, don't complain to their support team, you may get the same cantankerous tech that I got... :o

Jim

[GaryK]

I'm with you on this one, Pfui.

They use more bandwidth than all the other bots combined with the exception of Google. And I get so little traffic from them it doesn't even show-up on the pie charts I see each week showing which traffic came from what search engines.

So lately I've been wondering why I keep allowing MSN to crawl my sites.

I guess it's fear of what might happen at some point if I do deny them. That's irrational at best, but that's how it goes sometimes. I'm afraid to deny Microsoft. Plain and simple. And I'm sure they know most of us feel the same.