Welcome to WebmasterWorld Guest from 184.73.18.109

Forum Moderators: mack

MSN Spider is in!

They're building their own index - maybe

   
2:51 am on Dec 7, 2003 (gmt 0)

10+ Year Member



I saw the MSNbot go through my pages today (research.msn.com) was the bot name. Maybe they're building their own index with their own search technology!

Consider the possibilities of a microsoft search engine...they've got more funds than Google!

3:56 am on Dec 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Earlier...

[webmasterworld.com...]

Pendanticist.

6:07 am on Dec 7, 2003 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I think this is the original thread here @ webmasterworld:
[webmasterworld.com...]

And yep, lots of articles about how they are planning on building their own engine.

7:31 am on Dec 7, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Note that we're talking about two different user-agents here, msnbot and MSRBOT.

Jim

10:38 am on Dec 8, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



(Lotsa threads, quite true. Been in on many of them too. Also, my bad for not finishing my thoughts when I first posted.)

Because of all that 'history', I have to wonder why two technologies? Why two research centers? What could they have going on that requires what appears to be two seperate entities/factions?

I heard back from Eytan Seidman regarding how this new one only crawls one very remote, very obsure file as they are experimenting and will never crawl a site completely. Huh? One file and an obscure one at that? <shrug>

http://search.msn.com/msnbot.htm
http://research.microsoft.com/research/sv/msrbot/
http://research.msn.com? <-mebe markis00 could provide UA string?

Pendanticist.

9:38 pm on Dec 8, 2003 (gmt 0)

10+ Year Member



it was the msrbot
7:47 am on Dec 10, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>it was the msrbot

That's not what you mention in your initial post.

If you could provide the UA string, we might could get some answers. :)

Thanks.

Pendanticist.

1:30 am on Dec 22, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Just a heads-up that msnbot/0.11 is out and about, and obeying robots.txt (from what I see). It looks like a few issues seen in MSNBOT/0.1 have been fixed.

204.95.**.*** - - [21/Dec/2003:07:59:37 -0500] "GET /blah.html HTTP/1.0" 200 1231 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"

Jim

11:11 pm on Dec 22, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I opened a site, a new one, and saw MSN bot crawling to before Googlebot, In about a day or so when it went alive!
And the site didn't have a single link from an external site at that time, I wonder how it finds these pages.

This MSN bot is great technology.

Lots of people get confused between MSN bot and MSR bot, just so if you're confused:
MSNbot - Microsoft Network Bot - Used for the search engine currently in development
MSRbot - Microsoft Research Bot - Used for the Microsoft Research project or something, they have labs at many locations.

Sid

11:04 am on Jan 4, 2004 (gmt 0)

10+ Year Member



I've had visits from msn bot on 2 new sites, but have not touch any older sites that I have.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month