homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

This 75 message thread spans 3 pages: < < 75 ( 1 [2] 3 > >     
Microsoft MSN Bot Live in the Wild!
Microsoft is Crawling

10+ Year Member

Msg#: 26 posted 2:02 am on Jun 17, 2003 (gmt 0)

Look who's all over my sites!

"MSNBOT/0.1 (http://search.msn.com/msnbot.htm [search.msn.com])"

I've checked the IP - it's legitimate, Microsoft is crawliing the web!

IP 131.107.137.xxx



WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 10:47 am on Jun 18, 2003 (gmt 0)

WiseNut ran on Windows machines didn't it?


WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 26 posted 10:53 am on Jun 18, 2003 (gmt 0)

So did HotBot. ;)


WebmasterWorld Senior Member vitaplease us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 26 posted 11:06 am on Jun 18, 2003 (gmt 0)

Microsoft has published some incredible research in collaboration with a university in Beijing, China.

jeremy - did a site search for Beijing and had no joy - so how about posting those links again :-)

The Chinese connection?

check the names of the authors


10+ Year Member

Msg#: 26 posted 11:32 am on Jun 18, 2003 (gmt 0)

Dear martinibuster

Sorry, you are right, I went off topic. Or at least it may look like it. What I tried to convey in a few words is a whole flurry of feelings that I have when hearing about MSNBOT. And you are right, it was a knee jerk reaction to post myself a bit unexplained.

But it was not intended to be off topic. My feeling is that a MSNBOT is actually not good competition. I recall MS pushing out Netscape Navigator with unfair (bully) methods (and that's the opinion of the DOJ, not only mine). I can see MS buying a lot of different companies and utilise synergy effects (which in itself is not bad). Rights to photos and movies and other media continue to culminate in things such as Encarta (once upon a time born out of a forth class encyclopedia company). MS still tries to unthrone Apache with its IIS (or whatever it is called today). The last few OS installments were no substantial change in quality, but a bundling of third party software packages (movie maker, media player etc) that one could have installed themselves if needs be. There are still better and more stable OSes around. MS failed at positioning MSN as an ISP several times and so now they position it as a portal which no doubt will one day turn into a micropayment infoportal. MS is trying to monopolise the handling of usernames and passwords via its passport initiative, one step closer to dominating all online payments and being able to keep track of most transactions or even simple website visits. MS plays the friendly open source card with regards to XML and web services, but if you look at biztalk you'll see that they use that same dangerous mix of open source / open standards and proprietary bits that they used with Java ("Sure, that is great, but if you use our product you will have an enormous speed enhancement on all windows machines. And as you know there are over 90 percent of those around!"). MS was the first company to build into its software a forced registration and even re-registration if you decide to substantially upgrade your computer.

I am scared! I am scared of too much power in one hand. People here on the boards talk about the power Google has in listing or not listing your site, how Google can make or break a business (and I don't like that power, too!). But you ain't seen nothin' yet! If we allow (hence the robots.txt file I wrote about) MS to not only step into the search engine world, but to build up a market share of say 25 percent, then MS will use its dominance in other areas and push competitors out of the market. Not because they are better, but because they can. And after the competitors are pushed out, can you tell me how the future will look like then?

martinibuster, I have to strongly disagree with one point you made: MS is definitely not a bunch of dorks when it comes to marketing, quite the opposite! I have seen many better products or services in the many areas that MS competes in by now, but MS makes it sooo easy for people to go MS that they don't know what they are missing. Just recently I installed OpenOffice on a friend's machine and she said "Wow, this is free software? I don't have to pay for it? How is that possible?". And I also recall the MS propaganda paper Communique, which during those years of Department of Justice troubles not once spoke about MS being the baddies, but insulted the DOJ as well as the government and anyone open its mouth against MS. There is no doubt in my mind, MS is a bully, propagandist and demagogist and will never change, no matter if they spend billions on bringing MS-equipped machines to third world countries or not (Peru refused their offer and instead goes open source. Result? MS would love their government toppled!).

Back to the topic of this emotional post: I for my part will try to stop - with my very very limited means - the MS competition in the search engine world. Not because I don't like competition, but the exact opposite! I love competition and I can see that two years of healthy competition would be followed by centuries of dominance that can not be broken into anymore if MS gets its way - again!

And I still invite anyone to do the same before it's too late!

End of speech, I'll get off my soap box now...

Oh, and martinibuster, this is in no way supposed to be an attack on you, just my view of things to give something to the discussion.


10+ Year Member

Msg#: 26 posted 2:35 pm on Jun 18, 2003 (gmt 0)

Heh heh. The Gates Foundation. How fast can you say tax breaks?
I don't have a problem with their donations (many people will benefit) but they are not totally motivated by egalitarianism, there are selfish reasons for them.

As for the MSN-Bot, good on them. People will not use it unless it is a good product.

I will be interested to see how competently the algo is put together. MS has a history of bad internal communication, if they have too big a team developing it, the whole thing will be a slow mess that doesn't deliver cohesive results. MS (in my opinion) has never done anything efficiently and a search engine relies on delivering fast, relevant results.

I will keep watching with interest.



10+ Year Member

Msg#: 26 posted 2:37 pm on Jun 18, 2003 (gmt 0)

there are selfish reasons for them.

$6 billion is selfish? thats not going to help the tax bill that much...!

jeremy goodrich

WebmasterWorld Senior Member jeremy_goodrich us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 26 posted 4:56 pm on Jun 18, 2003 (gmt 0)

Sure, WiseNut ran on Windows boxes. And how fast does it work?

Really, if ANY corporation expects their web applications to scale, they should pay close attention to if / when MSN releases their own search engine.

*if* it's NOT running on a windows platform, then that is basically Microsoft agreeing with the group here: Windows CAN'T scale to meet the rigorous demands of information retrieval on the web.

Sure, I like my windows 2K at work, and my windows XP at home - but a few years ago, there was a moderator here who found a Microsoft server on one of their IP's, running Linux - and guess what? The thing had been stable (up & running) for well over 100 days.

Try doing that with a Windows server, more than likely, it won't happen.

Though if they *could* get their stuff together, as I mentioned before, they have adapted faster to the right kind of technology to power their products than any other company in the US. At least, imho.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 5:00 pm on Jun 18, 2003 (gmt 0)

I did hear that MS recently purchased some licensing from SCO.


10+ Year Member

Msg#: 26 posted 7:49 pm on Jun 18, 2003 (gmt 0)

Today MSN referrals are actually rivaling Google on a couple of my sites. But these are INK serps not the result of MSN spidering. I'm seeing the MSN spider deepcrawling but when I take several specific referrals from MSN out of my log then run the query through the MSN and INK searches, the serps are identical.

But with G seeming to be stumbling over itself, it is nice to pick up the traffic from MSN.


WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 26 posted 8:57 pm on Jun 18, 2003 (gmt 0)

out of interest keyplr, how many pages roughly was that? - richmc

43 out of 108.

Abided by robots.txt and didn't request any of the disalowed pages or directories.


WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 26 posted 9:23 pm on Jun 18, 2003 (gmt 0)

> Today MSN referrals are actually
> rivaling Google on a couple of my sites.

That's not something I would brag about ;-)


WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 10:43 pm on Jun 18, 2003 (gmt 0)

My apologies for the off-topic posts, Brett.

Still no sign of the thing here anyway. Maybe Bill doesn't like me.


WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 26 posted 11:16 pm on Jun 18, 2003 (gmt 0)

>>Maybe Bill doesn't like me.

I can tell about if Bill doesn't like you. 10 k US.
I can tell about if Google doesn't like you. 100 k US.
I can tell about if your mother doesn't like you. 100 000 k US.

Why is 950 % more.

Any more questions?


10+ Year Member

Msg#: 26 posted 12:18 am on Jun 19, 2003 (gmt 0)

This might be useful info:

Check out Dave Winer's Anonymous Source on MSNBot [scriptingnews.userland.com].


  • Microsoft's informal bid to buy Google, refused
  • Microsoft decides to build their own search infrastructure to compete with Google
  • Microsoft plans to further bind this search infrastructure to IE and the Longhorn version of Windows.


WebmasterWorld Administrator martinibuster us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 26 posted 1:10 am on Jun 19, 2003 (gmt 0)

and in Windows Longhorn (the Search function in the shell will have an Internet option that will go to MSN by default).


This is what I've been saying for a long time... MS hasn't said so explicitly, but it's implicit in all the articles and "leaks" about Longhorn I've read. The media hasn't realized the importance of this so they haven't thought of asking them about it.

This is tied to the new file system being developed for Longhorn. Everything is integrated: from IE to Searching your offline files to searching online.

I suspect what they want to do is own search directly from the desktop, from the moment you press the "on" switch of your computer.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 1:19 am on Jun 19, 2003 (gmt 0)

This just may be the incentive google needs to create that toolbar for the mac.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 11:06 am on Jun 19, 2003 (gmt 0)

Well, I guess I need to change my name to ‘egg-on-my-face’!


My guess is that you will be able to either turn off the longhorn search function, no telling how hard it will be to do that, or be allowed to change to another default search engine. MS has had too many anti-trust cases already not to be ready for yet another one.

I have a couple things I am wondering about.

1. I know of no one that has banned googlebot, but lots that have banned MS’s IP address ranges. So, how successful will this endeavor be in the short term when a lot of pages they crawl must be getting 403’s? Are they also using a directory or storing Ink results to use in the short term until everyone MUST un-ban MS IP’s? Will they add more IP’s for the bot as time goes on to try to index all kazillion pages that exist on the internet?

2. How related is MSN to MS as a whole? Also, How related is MSNBC to MS as a whole? I don’t know what kind of relationships there are with these companies. How much information is shared between the different parts of the whole?


10+ Year Member

Msg#: 26 posted 4:52 pm on Jun 19, 2003 (gmt 0)

MSN DEEP crawled one of my clients sites this morning!

12:32:53 MSNBOT/0.1_(http://search.msn.com/msnb Homepage CO 1
3:31:57 MSNBOT/0.1_(http://search.msn.com/msnb Infolookup CO 36
3:46:57 MSNBOT/0.1_(http://search.msn.com/msnb AddLink CO 1


10+ Year Member

Msg#: 26 posted 7:31 pm on Jun 19, 2003 (gmt 0)

An article on this subject...



10+ Year Member

Msg#: 26 posted 9:47 pm on Jun 19, 2003 (gmt 0)

Additional Press Releases

Is MSN gunning for Google? by CBS Marketwatch [cbs.marketwatch.com]

By Frank Barnako , CBS.MarketWatch.com
Last Update: 10:38 AM ET June 19, 2003

WASHINGTON (CBS.MW) -- Microsoft "searchbots" are reportedly scouring the Web to build a database and search tools that could compete with Google.

he software is compiling hyperlinks, Web pages and other documents as part of an effort that began in February to bolster the capabilities of MSN Search, according to ZDNet. Currently MSN has licensing agreements to use search tools from Inktomi ( YHOO :news ,chart ,profile ) and Overture Services ( OVER :news ,chart ,profile ). "At this point we are interested in developing the technology in-house," Lisa Gurry, MSN's group product manager, told Cnet...


MSN search bot a glimpse of ambitions By News.com [news.com.com]

By Jim Hu
Staff Writer, CNET News.com
June 18, 2003, 6:27 PM PT

In preparation for unveiling its own algorithmic search engine, Microsoft's MSN has quietly launched software that will index Web sites, a move that raises questions about MSN's relationship with Yahoo subsidiary Inktomi.

Called MSNBot, the software scours Web sites and collects hyperlinks and documents. The software is part of MSN's effort to challenge Google by revitalizing MSN Search with its own algorithmic search technology...


MSN gets on search bandwagon by News.com [news.com.com]

By Stefanie Olsen
Staff Writer, CNET News.com
February 11, 2003, 11:54 AM PT

Web portal MSN is testing a new search service that touts faster, tidier results, in what is the latest development in a fast-moving contest to help people find what they're looking for online.

The Microsoft-owned division launched a test site for MSN Search on Tuesday that omits banner advertisements and a large navigation column for the MSN network, freeing up more room on the page to display query results. The new site also says that it can display results 65 percent faster than before...


10+ Year Member

Msg#: 26 posted 1:19 am on Jun 20, 2003 (gmt 0)

I'm seeing a robot called "MSIECrawler" in my server logs. Who is this? I have no ip info.


10+ Year Member

Msg#: 26 posted 1:30 am on Jun 20, 2003 (gmt 0)

msnbot has been all over my sites, today i got an email from:


within the email it sayes

"We discovered this error during our normal course of website content checking for one of our search engine clients"

Anybody else get one of these after being visited by MSNbot?

Meanwhile, i'm working on checking all my links.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 1:22 pm on Jun 20, 2003 (gmt 0)

I just noticed something when surfing through a german magazine that has a cooperation with msn. If you go to e.g.


and doubleclick on one of the words in the text a new windows opens and a search on msn for the word you doubleclicked on is loaded. For example place your cursor over the word "Beckham" and doubleclick and a search on msn for Beckham is started.
Has anyone noticed anything like this before? So far I have seen this nowhere else but if something like this was integrated into Internet Explorer Google and any other search engine could close down in no time.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 26 posted 5:13 pm on Jun 20, 2003 (gmt 0)

Brett, regarding your question about whether MSNBOT was only on 131.107.137.xxx, we're discussing the actual IP addresses over here [webmasterworld.com] in the Search Engine Spider Identification forum.


10+ Year Member

Msg#: 26 posted 6:40 pm on Jun 20, 2003 (gmt 0)

My apologies if this has been written before (haven't read through this thread as diligently as I should): could the timing of MSNbot have anything to do with Google's much-anticipated IPO? I don't mean the whole reason behind doing the SE (I'm thinking that's more about cashing in on the rise of PPC), I mean, if MS had a mole inside Google or some kind of advance notice of the imminent IPO, rolling out MSNbot immediately before would be a nice "dirty trick" to take some of the wind out of the IPO's sails. (Or am I just paranoid?)


10+ Year Member

Msg#: 26 posted 2:51 am on Jun 21, 2003 (gmt 0)


I have been getting those emails for two months now. I just thought they were junk mail.


Msg#: 26 posted 6:03 pm on Jun 22, 2003 (gmt 0)

darn it...

I've just checked my last week log and I got over 2,000 hits from MSNbot for a site that is less than 1000 pages.

What you guys think? Ban MSNbot?


WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 26 posted 3:39 am on Jun 23, 2003 (gmt 0)

>So did HotBot.

I have more computing power at my finger tips right now at WebmasterWorld than all of HotBot. It runs on a couple of older boxes (used to be a small network of p2's running nt) - it's a proxy server to Ink and now the other engines it uses - easy peasy.

Wisenut had at most 16 boxes running 2000 and access - total off the shelf. They were lucky to do 100k searches a day. eg: the same thing under ASPSeek (*nix) would have taken one box and probably been a better index.


Msg#: 26 posted 5:59 am on Jun 23, 2003 (gmt 0)

What about these results obtained in 2000:

"In support of the unprecedented scalability and performance gains of Microsoft® Windows 2000, [Bill] Gates announced a new world record Transaction Processing Performance Council (TPC) benchmark number of 227,079 tpmCs for Windows 2000 Advanced Server and Microsoft SQL ServerTM 2000 running on 12 Compaq eight-way ProLiant systems. This new world record is almost twice as fast as the nearest competitor, at one-fourth the price/performance. This system is so powerful that it is capable of handling the total number of retail e-commerce transactions in 1999 in only 48 hours."

"Gates also cited independent testing by Ziff-Davis Labs, which showed the specific reliability benefits of Windows 2000 over earlier generations. According to the test, with constant and intense use in the 90 workdays that Ziff-Davis Labs ran the test, Windows 2000 never failed. In comparison, Windows 95 had to be rebooted every 2.1 days and Windows NT® Workstation 4.0 had to be rebooted every 5.2 days." [microsoft.com...]


10+ Year Member

Msg#: 26 posted 9:46 am on Jun 24, 2003 (gmt 0)

What's all this anti-Microsoft feelings about?
Don't get me wrong, I'm not going to make a case for Microsoft here, but I thought this was a forum about search-engines, not a religious discussion!?
MSNBot has been all over my site in the last couple of weeks, and I'm happy for it. The more competition there is between SEs, the better the results are going to be, the easier user's are going to find our sites.
And, be fair, Google can use some good competition at the moment! Maybe then they'll get their act together and go back to the quality they used to have.


10+ Year Member

Msg#: 26 posted 3:19 pm on Jun 24, 2003 (gmt 0)

Hi Driesie!

Sorry, but I seem to fail to see any anti-microsoft sentiment here... I read a few comments that questioned MS's ability to run a search engine of the size that they plan on their own operating systems due to lack of scalability or uptime stability, I read posts that complained that their bot seems to request pages quite often and I read of concerns what may happen if MS "owns" all searches via the already predominant interface called Internet Explorer and directly via the Windows operating systems.

You are right, Google should shape up to prepare for the competition from any side, not only MS. But will MS really bring competition to the game? Could it not be that MS is a money-hungry company that may plan the following: Get a foothold in the SE market, then use Internet Explorer and the OS to direct more searches to MSN (if possible all searches. give very cryptic instructions on how to change default searching behaviour, so that no one ever attempts to change. that'll take care of anti-trust laws...). Then tell all webhosting companies that a new add-on is available for MS Internet Information Server (IIS) that allows better indexing of only IIS sites via MSNBot.

And once about 80 percent of all searches run via MSN we'll change to paid inclusion, overnight. And the money will flow.......

Well, I might be seeing ghosts here, but I might as well see an ugly possible future, hmm?

Competition yes, but MS is no competition, it's a bully using bully tactics, in the end stifling competition! Or when were you last asked what operating system you'd like on your new computer...


This 75 message thread spans 3 pages: < < 75 ( 1 [2] 3 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved