MSRBOT With Abovenet IP address

Is this legit

[dataguy]

I've been getting hit with a bot with a user agent string of 'MSRBOT (http://research.microsoft.com/research/sv/...'

The bot isn't coming from one of Microsofts normal IP address ranges, rather it's coming from an IP address registered to Abovenet in San Jose.

Does anyone know if this is legit? As it stands now, my site is blocking access since it doesn't recognize the IP as being legit.

Thanks for your help.

[wilderness]

Here's an old thread.
[webmasterworld.com...]

It helps if you include the full line from your visitor logs, otherwise it's impossible to determine what IP range your referring to.
When doing so the forum reqiures that you obscure the class D with X's.
If the numer is a single digit, one x
doubde digit, two x's
Triple digit, three x's.

(as a side note; in most instances the forum requirement doesn't defeat much as the majority of IP ranges are either in the Class B or C).

edite by wilderness:

Here's link to MSN and their explantion:
[research.microsoft.com...]

here's a google
[google.com...]

[edited by: wilderness at 1:23 am (utc) on Oct. 13, 2006]

[GaryK]

Hi DataGuy,

MSRBOT
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
MSRBOT/0.1
MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)

These are the user agents I have on file with MSRBOT in them. None of them is a legitimate MS user agent insofar as I can tell.

I've had ten visits from the ua you mentioned. All of them on October 10, 2006. None of them were from valid MS addresses. I will not comment on what rDNS or IP Addresses they visited me from as it usually gets me into trouble with Dan.

[wilderness]

Oct 7th

Abovenet Communications, Inc ABOVENET-4 (NET-209-249-0-0-1)
Five Elements MFN-T326-209-249-11-0-28 (NET-209-249-11-0-1)

same IP range returned on the 9th.

BTW, I've had the following in my robots.txt for some time:
User-agent: MSRBOT
Disallow: /

Whether they are MSN or not?
They comply with robot's text and did on both occassions aboved.

[GaryK]

According to my source at MS (Chief Evangalist for IE7) if a bot isn't coming from a MS IP Address range it's not a valid MS bot.

EDIT: And just because it's coming through a MS IP Address range doesn't always mean it's a valid MS bot. Same thing with Yahoo whom I work closely with because they use my browscap file for their PHP-related stuff.

[edited by: GaryK at 2:56 am (utc) on Oct. 13, 2006]

[wilderness]

According to my source at MS (Chief Evangalist for IE7) if a bot isn't coming from a MS IP Address range it's not a valid MS bot.

My friend Gary (keep that in mind),
Don't believe a word those lying SOB's offer!

Microsoft has a history of crawling both unidentified and from non-MSN ranges since they began with their bots from their inception!
I'm not willing to spend useless time digging through forum archives to justify my statement.

I will however provide some saved links

[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]

[GaryK]

Of course you're my friend Don. There's never been any doubt about that.

I'm sure you're at least partially right about MS. My experiences with them, apart from their damned bugs in browscap.dll have always been good. I've never felt like I was being lied to. Maybe that's a mark of how good they are at it. But my guy always gets back to me and has helped me find answers to things related to user agents that I'd never be able to find on my own.

[wilderness]

I'm sure you're at least partially right about MS.

Am I'm not alone!
When MS first began spidering, I seen to recall a thread at SESID that approached THIRTY (30) pages.

[GaryK]

Fair enough. That's what makes this such a good site. We can all have different opinions and still get along. :)

[thetrasher]

Abovenet Communications, Inc ABOVENET-4 (NET-209-249-0-0-1)
Five Elements MFN-T326-209-249-11-0-28 (NET-209-249-11-0-1)

[spews.org ]:

five-elements/bitpushers/ings
(...)
Spamming direct and via hijacked proxies & spammer hosting

[jdMorgan]

I noted that although most of the IP range belongs to MS, not all of it does, and I was getting a "no hostname" response when querying MS's DNS.

So my conclusion was that this is what I call a "hole-dweller" -- a nasty weasel-like creature that lives in a (IP) hole very near a legitimate hunter, and feeds off the scraps. (No offense to real weasels, living or dead is intended.)

Jim

[wilderness]

Whether they are MSN or not?
They comply with robot's text and did on both occassions aboved.

[dataguy]

Thanks guys... I've had about 20,000 pages crawled by them in the past 24 hours and even if it is from MS I don't see any reason to help them with research, so I've blocked them.

[Mokita]

They comply with robot's text

They don't in one of our sites.

The first time I saw them I didn't have it disallowed in robots.txt and it asked for robots.txt, index page and one other page.

Second time it came by a few days later, I did have it disallowed but it retrieved the exact same three files.

[wilderness]

Mokita,
Maybe they just don't like Aussies ;)

Don

[Mokita]

Wilderness - I'm beginning to think that you don't like Aussies either! ;) :D

[wilderness]

I think every Yank outa own at least 2-3
Aussies and Kiwi's ;)

BTW, I'm just getting ready to archive a brief article and photo of the stop watch that was used for Lawn Derby's 1939 record at Cantebruy, NZ.

Previously archived the article and photo from the January 1939 announcement.

Have another brief and photo of an old-timer named Lawn Derby.

If your interested?
sticky me your email and I'll send.

Don

[BillyS]

I just noticed this one too:

Agent: MSRBOT (http://research.microsoft.com/research/sv/msrbot/)

209.249.11.3

Whois Record

Abovenet Communications, Inc ABOVENET-4 (NET-209-249-0-0-1)
209.249.0.0 - 209.249.255.255
Five Elements MFN-T326-209-249-11-0-28 (NET-209-249-11-0-1)
209.249.11.0 - 209.249.11.15