Welcome to WebmasterWorld Guest from 23.20.110.176

Forum Moderators: mack

Message Too Old, No Replies

Microsoft MSN Bot Live in the Wild!

Microsoft is Crawling

     
2:02 am on Jun 17, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:June 6, 2003
posts:67
votes: 0


Look who's all over my sites!

"MSNBOT/0.1 (http://search.msn.com/msnbot.htm [search.msn.com])"

I've checked the IP - it's legitimate, Microsoft is crawliing the web!

IP 131.107.137.xxx

2:15 am on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 7, 2001
posts:661
votes: 0


Others have seen the MS bot before, but I have not seen the information URL.

Previous threads
[webmasterworld.com...]
[webmasterworld.com...]

2:16 am on June 17, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14172
votes: 204


This is cool, good find. At last they have their bot faq up.
3:35 am on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 4, 2002
posts:1687
votes: 0

Read that 7 page thread, then went to http://search.msn.com/msnbot.htm

So it's decided that it's legit? It looks that way but I'm starting to get fuzzy...

4:30 am on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 2002
posts:1056
votes: 0


MSN results are actually looking better today than on G.
4:48 am on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 28, 2001
posts:1380
votes: 0


MSN results are actually looking better today than on G.

Are you part of the "trustworthy computing" campaign? :-)

Once I got past the pop-up window I found the reslts to be...well...I better stop before I say something I regret!

6:01 am on June 17, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14172
votes: 204


Once I got past the pop-up window...

dvduval, what url are you typing, and from what part of the world? I do not receive a pop-up.

Let me guess (and you be honest!) are you a Mac loyalist?

:) Y Y

8:09 pm on June 17, 2003 (gmt 0)

Full Member

joined:Sept 21, 2002
posts:246
votes: 0


Are they crawling with MSN bot because they wont be able to use inktomi soon - now that Yahoo owns ink?
8:54 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


Hey hey - so we are looking at 4 major players in the websearch game soon?
G -- ATW/AV/OV -- Y!/INK -- MS
8:59 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 28, 2001
posts:1380
votes: 0


Hey hey - so we are looking at 4 major players in the websearch game soon?
G -- ATW/AV/OV -- Y!/INK -- MS

I wonder how long it will take people to actually start searching the others again now that they are so conditioned to use Google.

I would guess Yahoo has the greatest chance of success, followed by MSN. I love ATW, but I question how easily it can become widespread.

9:01 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


Microsoft has published some incredible research in collaboration with a university in Beijing, China.

The interesting bit is that they were looking at other types of correlations than the ones that the other players look at to map 'relevance'.

Another interesting bit is that Microsoft is one of the first large US corporations to incoporate fuzzy systems into their products, and that China has the world's largest number of fuzzy scientists.

Google's technology is based on a probabilistic model which is a subset of fuzzy logic, as determined by 'probability & the hypercube' by a professor named Bart Kosko.

In my mind, better math = better engineering. If I were Google, I would seriously rethink my bias against non traditional systems & fuzzy logic, and toss the binary thinking that may constrain their ability to return optimal web results.

9:05 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 28, 2001
posts:1380
votes: 0


Jeremy,

When you talk of departing from binary thinking, do you mean displaying dynamic results that can vary on a page refresh?

9:55 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


Try reading through a bit of MS history / engineering / and their papers out of the Beijing University I mentioned (might have posted a link here, not sure).

Google in their research, continually reference probability, as the solution to deal with ambiguous or incomplete data. They have an incredible search engine, but at Stanford, even the current research on 'bettering PageRank' involves using a different approach.

I love their engine, however, when you read through their old papers, it's pretty clear what math school of thought they were following.

The stuff that MS has been doing for years, is only possible with fuzzy systems...applied to the web, and you get math in 3d instead of 2d.

10:01 pm on June 17, 2003 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38061
votes: 13


> MSN results are actually looking
> better today than on G.

Some days, there ought to be a license to compute ;) Put the keyboard down and step away from the computer. lol!

>131.107.137.xxx

So is it just that ip block? That is one for the htaccess list for sure.

10:09 pm on June 17, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 2, 2002
posts:274
votes: 0


I noticed some increases in MSN traffic the last few days and wondered myself what was going on.

1 www.google.com 43.0%

2 search.yahoo.com 19.7%

3 search.msn.com 18.9%

4 aolsearch.aol.com 6.1%

Two weeks ago...

1 www.google.com 43.5%

2 search.yahoo.com 32.2%

3 search.msn.com 11.3%

4 aolsearch.aol.com 4.3%

10:15 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 4, 2002
posts:1687
votes: 0


Well, so sign of the beast in the logs for yesterday... I'm in two ODP cats, the thing should show up soon I hope. As much as I detest Bill, I'd hate to be left out of the party.

Is anyone seeing it being particularly active yet?

10:30 pm on June 17, 2003 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38061
votes: 13


>Is anyone seeing it being particularly active yet?

Yes, they are seeking out SEO's to beta test (they call you - don't call them). However, the NDA has more pages than webmasterworld I believe.

The only question left is if it will run Windows or more probably on Unix or an IBM mainframe.

10:44 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 4, 2002
posts:1687
votes: 0


Man, I guess I'll have to read through a lot of archived threads...

So how will this work? Once they have a large enough index from the new bot, they dump Ink and go with their own data?

(NDA - National Dance Association. A fine bunch, by all reports)

11:07 pm on June 17, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 4, 2001
posts:84
votes: 0


I do not know if this has been noticed before, but MSN is looking for search engine developers. See Job Opportunities with MSN Search [search.msn.com].

I guess they could not buy Google so now...shudder the thought...

11:21 pm on June 17, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14172
votes: 204


The newly expanded MSN Search team is working on indexing the entire internet and returning best-of-class results to search queries.

Wow.

Think they'll do a PFI?

I wonder if internet search will be integrated with Longhorn.

IE is suspended [ecommercetimes.com] as a standalone product, and the next iteration will be as an integrated aspect of the Longhorn OS.

[edited by: martinibuster at 11:28 pm (utc) on June 17, 2003]

11:26 pm on June 17, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2001
posts:1609
votes: 0


My guess is that the index will be an integral part of the new "we don't have a browser bundled, because we don't call it a browser" OS.

Search integrated with Word...

Why change tactics now?

11:57 pm on June 17, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 19, 2003
posts:70
votes: 0


Great, a new robots.txt will go up right now :-)

User-Agent: MSNBOT
Disallow: /

...and then I wait for MS to not obey it. I just can't imagine them getting THIS right!

If you folks out there would like to help me: Simply use above robots.txt and post the first time the bot does not obey. I'd love to see that!

Sorry, but Bill has the rights to pictures of Marilyn Monroe and the Moon Landing, he really does not need to tell me what to find when I am searching the web!

And about losing a few bucks in say the next year because Bill has integrated the search into the OS: "They can take our bucks, but they can't take our FREEEEEEEDOM!"

2:59 am on June 18, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2003
posts:633
votes: 0


If I'm not mistaken, the search is already integrated into the OS. There's two ways that I know of to search the web using MSN straight from Windows :

1) Simply type your search term into the Internet Explorer address bar.

2) Start Menu -> Search -> On the Internet

-panic

3:27 am on June 18, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14172
votes: 204


Amadeus,

I have seen a number of your posts around, and I respect your opinion- But I have to question your last post.

Yes, they have the rights to a lot of photos. Yes they make a lot of money. Yes MS has a ubiqitous presence in computing and finance. Yes they're a bunch of dorks when it comes to marketing. Yes, because of their size and the abundance of MS Hacking Tools they're the target of every angry overprivileged teenage hacker...

But is this really a reason to be resentful of MS if they want to get into the web search business? I say no. It's not logical.

Think about this: Visit the Bill and Melinda Gates Foundation [gatesfoundation.org] web site and see how they've donated over SIX BILLION DOLLARS in 2003 to worthy causes like fighting aids in Africa, education, libraries, etc.

What has Apple done besides self serving donations of computer equipment to design schools in the 80's?

Has Google behaved as a responsible citizen of the world and given back even a fraction of a fraction of 1 percent to make life a better place for those less fortunate?

Please don't be confused: I am not positing Microsoft's charitable practices versus Apple or Google.

Perhaps this is a little off-topic but, with great respect for your opinion, so is your post. And that's the point of this post: It's important to keep to the topic.

I'm not discounting the value of healthy skepticism but your post goes beyong skepticism and, no disrespect to you intended, it delves into the realm of knee-jerk reactionism- which does nothing to further the discussion of a Microsoft Bot in the wild.

But I agree with you Mozart, they can take our bucks, but they can't take our freedom.

I applaud MS's foray into search because it has the potential to give Google some competition, and that can only be good for everyone because it will drive innovation.

[edited by: martinibuster at 3:40 am (utc) on June 18, 2003]

4:56 am on June 18, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Back to topic, they seem to have little adjustments to do about clustering. I have results from the sames domains repeated several times in first 3 SERP. Is it an clue they are mixing results from several sources?
7:17 am on June 18, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6535
votes: 114


Saw MSNBOT today for the first time. Crawled about half our pages, requesting robots first off.

8:53 am on June 18, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


How could MSN not have a search engine powered by MS technology? It must be surely embarrassing to them that their search results are powered by open source OSs.
10:19 am on June 18, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 15, 2002
posts:64
votes: 0


Saw MSNBOT today for the first time. Crawled about half our pages, requesting robots first off.

out of interest keyplr, how many pages roughly was that?

10:37 am on June 18, 2003 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38061
votes: 13


> search results are powered by open source OSs.

The first net engines showed up in 94, and there are historical firsts in db/se programs long before that.
And it has taken MS until 2003 to do something? The worlds largest software manufacture with 20k'ish programmers. hmmm speaks loudly of the ability of Windows to scale (lack thereof).

10:45 am on June 18, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 17, 2003
posts:384
votes: 0


Can you say that?
This 75 message thread spans 3 pages: 75