MSN Spider is in!

They're building their own index - maybe

[markis00]

I saw the MSNbot go through my pages today (research.msn.com) was the bot name. Maybe they're building their own index with their own search technology!

Consider the possibilities of a microsoft search engine...they've got more funds than Google!

[pendanticist]

Earlier...

[webmasterworld.com...]

Pendanticist.

[jeremy goodrich]

I think this is the original thread here @ webmasterworld:
[webmasterworld.com...]

And yep, lots of articles about how they are planning on building their own engine.

[jdMorgan]

Note that we're talking about two different user-agents here, msnbot and MSRBOT.

Jim

[pendanticist]

(Lotsa threads, quite true. Been in on many of them too. Also, my bad for not finishing my thoughts when I first posted.)

Because of all that 'history', I have to wonder why two technologies? Why two research centers? What could they have going on that requires what appears to be two seperate entities/factions?

I heard back from Eytan Seidman regarding how this new one only crawls one very remote, very obsure file as they are experimenting and will never crawl a site completely. Huh? One file and an obscure one at that? <shrug>

http://search.msn.com/msnbot.htm
http://research.microsoft.com/research/sv/msrbot/
http://research.msn.com? <-mebe markis00 could provide UA string?

Pendanticist.

[markis00]

it was the msrbot

[pendanticist]

>it was the msrbot

That's not what you mention in your initial post.

If you could provide the UA string, we might could get some answers. :)

Thanks.

Pendanticist.

[jdMorgan]

Just a heads-up that msnbot/0.11 is out and about, and obeying robots.txt (from what I see). It looks like a few issues seen in MSNBOT/0.1 have been fixed.

204.95.**.*** - - [21/Dec/2003:07:59:37 -0500] "GET /blah.html HTTP/1.0" 200 1231 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"

Jim

[sidyadav]

I opened a site, a new one, and saw MSN bot crawling to before Googlebot, In about a day or so when it went alive!
And the site didn't have a single link from an external site at that time, I wonder how it finds these pages.

This MSN bot is great technology.

Lots of people get confused between MSN bot and MSR bot, just so if you're confused:
MSNbot - Microsoft Network Bot - Used for the search engine currently in development
MSRbot - Microsoft Research Bot - Used for the Microsoft Research project or something, they have labs at many locations.

Sid

[thehittmann]

I've had visits from msn bot on 2 new sites, but have not touch any older sites that I have.