Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

Microsoft MSN Bot Live in the Wild!

Microsoft is Crawling



2:02 am on Jun 17, 2003 (gmt 0)

10+ Year Member

Look who's all over my sites!

"MSNBOT/0.1 (http://search.msn.com/msnbot.htm [search.msn.com])"

I've checked the IP - it's legitimate, Microsoft is crawliing the web!

IP 131.107.137.xxx


10:47 am on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

WiseNut ran on Windows machines didn't it?


10:53 am on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

So did HotBot. ;)


11:06 am on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member vitaplease is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Microsoft has published some incredible research in collaboration with a university in Beijing, China.

jeremy - did a site search for Beijing and had no joy - so how about posting those links again :-)

The Chinese connection?

check the names of the authors


11:32 am on Jun 18, 2003 (gmt 0)

10+ Year Member

Dear martinibuster

Sorry, you are right, I went off topic. Or at least it may look like it. What I tried to convey in a few words is a whole flurry of feelings that I have when hearing about MSNBOT. And you are right, it was a knee jerk reaction to post myself a bit unexplained.

But it was not intended to be off topic. My feeling is that a MSNBOT is actually not good competition. I recall MS pushing out Netscape Navigator with unfair (bully) methods (and that's the opinion of the DOJ, not only mine). I can see MS buying a lot of different companies and utilise synergy effects (which in itself is not bad). Rights to photos and movies and other media continue to culminate in things such as Encarta (once upon a time born out of a forth class encyclopedia company). MS still tries to unthrone Apache with its IIS (or whatever it is called today). The last few OS installments were no substantial change in quality, but a bundling of third party software packages (movie maker, media player etc) that one could have installed themselves if needs be. There are still better and more stable OSes around. MS failed at positioning MSN as an ISP several times and so now they position it as a portal which no doubt will one day turn into a micropayment infoportal. MS is trying to monopolise the handling of usernames and passwords via its passport initiative, one step closer to dominating all online payments and being able to keep track of most transactions or even simple website visits. MS plays the friendly open source card with regards to XML and web services, but if you look at biztalk you'll see that they use that same dangerous mix of open source / open standards and proprietary bits that they used with Java ("Sure, that is great, but if you use our product you will have an enormous speed enhancement on all windows machines. And as you know there are over 90 percent of those around!"). MS was the first company to build into its software a forced registration and even re-registration if you decide to substantially upgrade your computer.

I am scared! I am scared of too much power in one hand. People here on the boards talk about the power Google has in listing or not listing your site, how Google can make or break a business (and I don't like that power, too!). But you ain't seen nothin' yet! If we allow (hence the robots.txt file I wrote about) MS to not only step into the search engine world, but to build up a market share of say 25 percent, then MS will use its dominance in other areas and push competitors out of the market. Not because they are better, but because they can. And after the competitors are pushed out, can you tell me how the future will look like then?

martinibuster, I have to strongly disagree with one point you made: MS is definitely not a bunch of dorks when it comes to marketing, quite the opposite! I have seen many better products or services in the many areas that MS competes in by now, but MS makes it sooo easy for people to go MS that they don't know what they are missing. Just recently I installed OpenOffice on a friend's machine and she said "Wow, this is free software? I don't have to pay for it? How is that possible?". And I also recall the MS propaganda paper Communique, which during those years of Department of Justice troubles not once spoke about MS being the baddies, but insulted the DOJ as well as the government and anyone open its mouth against MS. There is no doubt in my mind, MS is a bully, propagandist and demagogist and will never change, no matter if they spend billions on bringing MS-equipped machines to third world countries or not (Peru refused their offer and instead goes open source. Result? MS would love their government toppled!).

Back to the topic of this emotional post: I for my part will try to stop - with my very very limited means - the MS competition in the search engine world. Not because I don't like competition, but the exact opposite! I love competition and I can see that two years of healthy competition would be followed by centuries of dominance that can not be broken into anymore if MS gets its way - again!

And I still invite anyone to do the same before it's too late!

End of speech, I'll get off my soap box now...

Oh, and martinibuster, this is in no way supposed to be an attack on you, just my view of things to give something to the discussion.


2:35 pm on Jun 18, 2003 (gmt 0)

10+ Year Member

Heh heh. The Gates Foundation. How fast can you say tax breaks?
I don't have a problem with their donations (many people will benefit) but they are not totally motivated by egalitarianism, there are selfish reasons for them.

As for the MSN-Bot, good on them. People will not use it unless it is a good product.

I will be interested to see how competently the algo is put together. MS has a history of bad internal communication, if they have too big a team developing it, the whole thing will be a slow mess that doesn't deliver cohesive results. MS (in my opinion) has never done anything efficiently and a search engine relies on delivering fast, relevant results.

I will keep watching with interest.



2:37 pm on Jun 18, 2003 (gmt 0)

10+ Year Member

there are selfish reasons for them.

$6 billion is selfish? thats not going to help the tax bill that much...!

jeremy goodrich

4:56 pm on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Sure, WiseNut ran on Windows boxes. And how fast does it work?

Really, if ANY corporation expects their web applications to scale, they should pay close attention to if / when MSN releases their own search engine.

*if* it's NOT running on a windows platform, then that is basically Microsoft agreeing with the group here: Windows CAN'T scale to meet the rigorous demands of information retrieval on the web.

Sure, I like my windows 2K at work, and my windows XP at home - but a few years ago, there was a moderator here who found a Microsoft server on one of their IP's, running Linux - and guess what? The thing had been stable (up & running) for well over 100 days.

Try doing that with a Windows server, more than likely, it won't happen.

Though if they *could* get their stuff together, as I mentioned before, they have adapted faster to the right kind of technology to power their products than any other company in the US. At least, imho.


5:00 pm on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I did hear that MS recently purchased some licensing from SCO.


7:49 pm on Jun 18, 2003 (gmt 0)

10+ Year Member

Today MSN referrals are actually rivaling Google on a couple of my sites. But these are INK serps not the result of MSN spidering. I'm seeing the MSN spider deepcrawling but when I take several specific referrals from MSN out of my log then run the query through the MSN and INK searches, the serps are identical.

But with G seeming to be stumbling over itself, it is nice to pick up the traffic from MSN.


8:57 pm on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

out of interest keyplr, how many pages roughly was that? - richmc

43 out of 108.

Abided by robots.txt and didn't request any of the disalowed pages or directories.


9:23 pm on Jun 18, 2003 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

> Today MSN referrals are actually
> rivaling Google on a couple of my sites.

That's not something I would brag about ;-)


10:43 pm on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

My apologies for the off-topic posts, Brett.

Still no sign of the thing here anyway. Maybe Bill doesn't like me.


11:16 pm on Jun 18, 2003 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

>>Maybe Bill doesn't like me.

I can tell about if Bill doesn't like you. 10 k US.
I can tell about if Google doesn't like you. 100 k US.
I can tell about if your mother doesn't like you. 100 000 k US.

Why is 950 % more.

Any more questions?


12:18 am on Jun 19, 2003 (gmt 0)

10+ Year Member

This might be useful info:

Check out Dave Winer's Anonymous Source on MSNBot [scriptingnews.userland.com].


  • Microsoft's informal bid to buy Google, refused
  • Microsoft decides to build their own search infrastructure to compete with Google
  • Microsoft plans to further bind this search infrastructure to IE and the Longhorn version of Windows.


1:10 am on Jun 19, 2003 (gmt 0)

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

and in Windows Longhorn (the Search function in the shell will have an Internet option that will go to MSN by default).


This is what I've been saying for a long time... MS hasn't said so explicitly, but it's implicit in all the articles and "leaks" about Longhorn I've read. The media hasn't realized the importance of this so they haven't thought of asking them about it.

This is tied to the new file system being developed for Longhorn. Everything is integrated: from IE to Searching your offline files to searching online.

I suspect what they want to do is own search directly from the desktop, from the moment you press the "on" switch of your computer.


1:19 am on Jun 19, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

This just may be the incentive google needs to create that toolbar for the mac.


11:06 am on Jun 19, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Well, I guess I need to change my name to ‘egg-on-my-face’!


My guess is that you will be able to either turn off the longhorn search function, no telling how hard it will be to do that, or be allowed to change to another default search engine. MS has had too many anti-trust cases already not to be ready for yet another one.

I have a couple things I am wondering about.

1. I know of no one that has banned googlebot, but lots that have banned MS’s IP address ranges. So, how successful will this endeavor be in the short term when a lot of pages they crawl must be getting 403’s? Are they also using a directory or storing Ink results to use in the short term until everyone MUST un-ban MS IP’s? Will they add more IP’s for the bot as time goes on to try to index all kazillion pages that exist on the internet?

2. How related is MSN to MS as a whole? Also, How related is MSNBC to MS as a whole? I don’t know what kind of relationships there are with these companies. How much information is shared between the different parts of the whole?


4:52 pm on Jun 19, 2003 (gmt 0)

10+ Year Member

MSN DEEP crawled one of my clients sites this morning!

12:32:53 MSNBOT/0.1_(http://search.msn.com/msnb Homepage CO 1
3:31:57 MSNBOT/0.1_(http://search.msn.com/msnb Infolookup CO 36
3:46:57 MSNBOT/0.1_(http://search.msn.com/msnb AddLink CO 1


7:31 pm on Jun 19, 2003 (gmt 0)

10+ Year Member

An article on this subject...



9:47 pm on Jun 19, 2003 (gmt 0)

10+ Year Member

Additional Press Releases

Is MSN gunning for Google? by CBS Marketwatch [cbs.marketwatch.com]

By Frank Barnako , CBS.MarketWatch.com
Last Update: 10:38 AM ET June 19, 2003

WASHINGTON (CBS.MW) -- Microsoft "searchbots" are reportedly scouring the Web to build a database and search tools that could compete with Google.

he software is compiling hyperlinks, Web pages and other documents as part of an effort that began in February to bolster the capabilities of MSN Search, according to ZDNet. Currently MSN has licensing agreements to use search tools from Inktomi ( YHOO :news ,chart ,profile ) and Overture Services ( OVER :news ,chart ,profile ). "At this point we are interested in developing the technology in-house," Lisa Gurry, MSN's group product manager, told Cnet...


MSN search bot a glimpse of ambitions By News.com [news.com.com]

By Jim Hu
Staff Writer, CNET News.com
June 18, 2003, 6:27 PM PT

In preparation for unveiling its own algorithmic search engine, Microsoft's MSN has quietly launched software that will index Web sites, a move that raises questions about MSN's relationship with Yahoo subsidiary Inktomi.

Called MSNBot, the software scours Web sites and collects hyperlinks and documents. The software is part of MSN's effort to challenge Google by revitalizing MSN Search with its own algorithmic search technology...


MSN gets on search bandwagon by News.com [news.com.com]

By Stefanie Olsen
Staff Writer, CNET News.com
February 11, 2003, 11:54 AM PT

Web portal MSN is testing a new search service that touts faster, tidier results, in what is the latest development in a fast-moving contest to help people find what they're looking for online.

The Microsoft-owned division launched a test site for MSN Search on Tuesday that omits banner advertisements and a large navigation column for the MSN network, freeing up more room on the page to display query results. The new site also says that it can display results 65 percent faster than before...


1:19 am on Jun 20, 2003 (gmt 0)

10+ Year Member

I'm seeing a robot called "MSIECrawler" in my server logs. Who is this? I have no ip info.


1:30 am on Jun 20, 2003 (gmt 0)

10+ Year Member

msnbot has been all over my sites, today i got an email from:


within the email it sayes

"We discovered this error during our normal course of website content checking for one of our search engine clients"

Anybody else get one of these after being visited by MSNbot?

Meanwhile, i'm working on checking all my links.


1:22 pm on Jun 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I just noticed something when surfing through a german magazine that has a cooperation with msn. If you go to e.g.


and doubleclick on one of the words in the text a new windows opens and a search on msn for the word you doubleclicked on is loaded. For example place your cursor over the word "Beckham" and doubleclick and a search on msn for Beckham is started.
Has anyone noticed anything like this before? So far I have seen this nowhere else but if something like this was integrated into Internet Explorer Google and any other search engine could close down in no time.


5:13 pm on Jun 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Brett, regarding your question about whether MSNBOT was only on 131.107.137.xxx, we're discussing the actual IP addresses over here [webmasterworld.com] in the Search Engine Spider Identification forum.


6:40 pm on Jun 20, 2003 (gmt 0)

10+ Year Member

My apologies if this has been written before (haven't read through this thread as diligently as I should): could the timing of MSNbot have anything to do with Google's much-anticipated IPO? I don't mean the whole reason behind doing the SE (I'm thinking that's more about cashing in on the rise of PPC), I mean, if MS had a mole inside Google or some kind of advance notice of the imminent IPO, rolling out MSNbot immediately before would be a nice "dirty trick" to take some of the wind out of the IPO's sails. (Or am I just paranoid?)


2:51 am on Jun 21, 2003 (gmt 0)

10+ Year Member


I have been getting those emails for two months now. I just thought they were junk mail.


6:03 pm on Jun 22, 2003 (gmt 0)

darn it...

I've just checked my last week log and I got over 2,000 hits from MSNbot for a site that is less than 1000 pages.

What you guys think? Ban MSNbot?


3:39 am on Jun 23, 2003 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

>So did HotBot.

I have more computing power at my finger tips right now at WebmasterWorld than all of HotBot. It runs on a couple of older boxes (used to be a small network of p2's running nt) - it's a proxy server to Ink and now the other engines it uses - easy peasy.

Wisenut had at most 16 boxes running 2000 and access - total off the shelf. They were lucky to do 100k searches a day. eg: the same thing under ASPSeek (*nix) would have taken one box and probably been a better index.


5:59 am on Jun 23, 2003 (gmt 0)

What about these results obtained in 2000:

"In support of the unprecedented scalability and performance gains of Microsoft® Windows 2000, [Bill] Gates announced a new world record Transaction Processing Performance Council (TPC) benchmark number of 227,079 tpmCs for Windows 2000 Advanced Server and Microsoft SQL ServerTM 2000 running on 12 Compaq eight-way ProLiant systems. This new world record is almost twice as fast as the nearest competitor, at one-fourth the price/performance. This system is so powerful that it is capable of handling the total number of retail e-commerce transactions in 1999 in only 48 hours."

"Gates also cited independent testing by Ziff-Davis Labs, which showed the specific reliability benefits of Windows 2000 over earlier generations. According to the test, with constant and intense use in the 90 workdays that Ziff-Davis Labs ran the test, Windows 2000 never failed. In comparison, Windows 95 had to be rebooted every 2.1 days and Windows NT® Workstation 4.0 had to be rebooted every 5.2 days." [microsoft.com...]


9:46 am on Jun 24, 2003 (gmt 0)

10+ Year Member

What's all this anti-Microsoft feelings about?
Don't get me wrong, I'm not going to make a case for Microsoft here, but I thought this was a forum about search-engines, not a religious discussion!?
MSNBot has been all over my site in the last couple of weeks, and I'm happy for it. The more competition there is between SEs, the better the results are going to be, the easier user's are going to find our sites.
And, be fair, Google can use some good competition at the moment! Maybe then they'll get their act together and go back to the quality they used to have.
This 75 message thread spans 3 pages: 75

Featured Threads

Hot Threads This Week

Hot Threads This Month