homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

This 64 message thread spans 3 pages: 64 ( [1] 2 3 > >     
MSN Search Hits Full Time Almost 100% of the Time now
out of beta

 6:30 am on Jan 22, 2005 (gmt 0)

Continued from:

woop01, thanks for posting your story about robots.txt
In 2 years, Yahoo, Google, Teoma (Ask), and Mama never excluded a site if it did not have a robots.txt file. The assumption seemed to be no robots.txt indicated permission to crawl everything. As I have posted on here many times, several pages of our site have been the #1 results for 2 years on all those SEs on 2 -4 keyword searches, with no robots.txt file. Not a single one of our pages shows up anywhere in MSN beta. So, it sounds like you MAY be describing our problem.

I just added a robots.txt to our root to see if THAT might be MSN beta's problem with listing our site. The file contains some comments and:
User-agent: *

If someone reading this is not familiar with robots.txt syntax, there are examples and a robots.txt file checker here:


Make sure you use an editor that breaks lines Unix style (line feed w/o carriage return). The Notepad editor, like most Windows platform tools, adds a carriage return with the line feed which won't work.

Sorry for the delay on responding to this part of the thread. I just want to quickly clarify this. MSNBot will support both and either. That is, you can use a carriage return and line feed or just one of those.

Additionally, like other crawlers we will crawl and index a site that has no robots.txt.

- msndude (msd)



 6:24 pm on Jan 22, 2005 (gmt 0)

For some reason many of the sites our company looks after are struggling to show much more than a few pages in beta even though they are being deep crawled. What's more when a specific search is entered it is usually the top level index page that is shown as opposed to the more relevant second level page even though it is in the database... why?


 7:28 pm on Jan 22, 2005 (gmt 0)

What's more when a specific search is entered it is usually the top level index page that is shown as opposed to the more relevant second level page even though it is in the database... why?

Because this is still a beta (I hope).

I am experiencing the same thing, the search results are pointing to a weaker page or a page that points to the correct page.

The Contractor

 7:55 pm on Jan 22, 2005 (gmt 0)

I am asking those who have posted here on this thread saying that MSN results are excellent to ....

I'll answer that. They are using the same criteria as you set forth - only the opposite. Your site(s) not included or ranking so they are censoring and crooked. Their sites are included and ranking well so it's the best SE out there.

Problem is that criteria is flawed big-time. You don't base/judge SE results on the criteria of your site. You also can not judge a SE by one search term here and there. You would have to use a large number of terms, say the top 10,000 terms searched for in a day and then run through all of the top SE's and have independent analysis. You also cannot tell the persons intent performing the search on many phrases or terms and neither can a SE.
Sorry, but I think they are doing OK for how long they have been indexing. There have been quite a few SE's over the years that never had results as good as what's showing now (and they had years to perfect them).


 1:53 am on Jan 23, 2005 (gmt 0)

csnet - frankly the results of both Google and Yahoo are currently poor and could be significantly better. If you ask me, both should carry the "beta" tag because they need improving!.

I think you are just miffed because you have a site ranking well in Google but not MSN. Frankly its time to stop moaning and time to look at ways to improve your content and website so that you do feature well in MSN also. MSN is NOT Google - thank God!

Also, at least with MSN you have a chance to improve you MSN SEO of your site, rather than in Google where many, many websites dont get that chance due to the sandbox! Think yourself lucky!

fleapit - you assume wrong that only top level is indexed. I have a number of pages that list well. Of these pages what i would say is that they are rich in content, are updated on a regular basis and have correct titles and quality links etc. Content is King on MSN.


 2:44 am on Jan 23, 2005 (gmt 0)

RichTC, this is not about improving content. There are too many content rich sites that are excluded from MSN beta results for "lack of content" to be a reason for excluding sites.

How many times do I have to to post that our site has more current and historical content about this actress than any other site on the Internet?

Our site is not just the #1 site for this actress 2 word keyword search on Google, but also on Yahoo, Teoma, Ask, Mamma. I have not mentioned it before but it is also #1 on Alexa, Alta Vista, and Excite. I went to a 3rd party site that has links to over 20 search engines. We are always #1. ONLY MSN beta excludes our site and dozens of other sites done by others in this genre.

Because MSN does not disclose how it rates sites, none of us can do anything except speculate about what causes a site to be included or excluded in MSN SERPs unles MSNdude posts something to set us straight about a particular item. :-)

As I mentioned, we provide links to 3rd party sites that we believe would interest our visitors. For all I know, one of these sites may link to a site that links to a site that does not meet MSN's criteria, so every site that links to that site even indirectly is dropped from MSN SERPs. But this is speculation because MSN is not disclosing it's methods. We certainly meet or exceed all the suggested criteria published in the MSN beta help pages.

As in the buy oranges in Korea example provided by roycerus above, MSN search logic must be seriously flawed to return such riduculous SERPs.

We could try dropping all links to 3rd party sites and hope that MSN will include us in the SERPs after that, but our site would be less useful to our visitors. In addition, the reasonable use of link exchanges make the Internet more useful. Can you imagine an Internet where the ONLY way to find a new site is through a SE? Perish the thought! lol


 5:16 am on Jan 23, 2005 (gmt 0)

I think the search engines should tell us how they rank our sites.

Then they would have good search results every time if we knew how to optomize for them.

If everyone knew the "secret" then it would just come down to link popularity and sites would be ranked accordingly.

Link popularity should be based on the amount of links pointing to a page and not Page Rank which is easy to manipulate by buying links from high PR sites. (Or was easy?).


 9:48 am on Jan 23, 2005 (gmt 0)

If you search google for buy oranges in korea now, number one slot is this thread discussing the bad MSN results for buy oranges in korea.



 10:45 am on Jan 23, 2005 (gmt 0)

An opportunity to corner the Korean orange market then...

There are loads of example of good and bad serps in msn, but so what. If it's generally good people will use it, if not they won't. It will never be perfect and neither will any of the others.

I imagine there are very few sites that are no. 1 on the three main se. Anyone that's getting 2 out of 3 is doing pretty well in my opinion.


 12:08 pm on Jan 23, 2005 (gmt 0)

<<If it's generally good people will use it, if not they won't. It will never be perfect and neither will any of the others>>

I don't think you can try and lump this disaster in with anything else or even try to compare it with any REAL search engine, this is the worst case of "giganticstuffupitis" I've ever seen.

Google or Yahoo have NEVER been this bad , in fact I can't think of any SE that has been this embarassingly awfull.

Heads will roll at MSN.


 2:42 pm on Jan 23, 2005 (gmt 0)

truly..reading this crap is sickening. it's painfully obvious that we have two people here bickering. One isn't ranking well and is complaining (sort of rightfully so) The other is OBVIOUSLY ranking well (I don't care if you deny it, any normal person browsing the serps wouldn't make the assertions that you are making). I'll say that I have several sites doing very well in msn BETA, and i'm still going to come and say that the engine in its current state sucks the big one. Don't get me wrong it has potential. But currently the serps are a mess. Like I said b4 in either this thread or another, while I enjoy having a good ranking it means little to me when I see myself surrounded by crap. I have searched a lot of things on the new msn as far as testing goes using totally different phrases than I would ever target for (this was the only way I figured I could avoid bias). While for some searches (honestly, in the big picture they're few and far between) they are very good, but for most it's really something awful. They have to know about the spam problems first off, Hell I had an email conversation with a developer about it. Not to mention sometimes the sites at the top, I'm sure, don't even expect to be there. They're not even spamming it's just a poorly applied algo. In short, they still have a long way to go before this thing is close to ready for official release. Even then i'm not going to get too excited. I've been near the top of their serps for several times, with both the new and old results. The traffic they send is mediocre at best and their conversion is so-so. I find google sends the most traffic, but Yahoo's traffic converts the best.


 3:52 pm on Jan 23, 2005 (gmt 0)

Mikec - I genuinely dont see the prolem you do with msn. Frankly im using the search and am finding that in the majority of cases the results returned are better than both Google and Yahoo.

In the case of Google almost every search i do these days has at least 3/4 directory sites in the top ten or sites returned that have bugg@r all content.

What i would say is that without any doubt google favours older sites over new - Where as i believe that MSN treats all sites as equal from the starting point and then looks at content Oh and MSN has no sandbox (yet!) .

Csnet - What you say is interesting but i dont want to sound cocky but i reccon if i looked at your actress site i would be able to quickly see why you are not listing in MSN. If you have the content and your pages are titled correctly and you have quality links to the actress name, your site should be listing well for those key words but i guess it depends on what keywords you are trying to list under in relation to the content of the page in question. I find that with Google once a site ranks well due to anchor links they list it for almost anything, i see this often for old sites.

I have a site with a section dedicated to "insurance Jobs" it has links to it with that Anchor text, its very rich in content and has over 10,000 Insurance jobs in it - so i would say that should be relevant to that search term in google. The section is 9 months old. Currently in Google it lists after sites relating to Insurance Quotes and buy Life insurance for an "Insurance Jobs" search term, so i could spend all day telling you how cr@p google is about bringing back results relevant to search terms couldnt i?.

Bottom line is that, as i see it Google is still based on the principle that Links are king V MSN which is clearly about content first.

Two different engines with different algos. It is possible to crack all three search engines but i havent done it myself yet, but a few others have


 3:54 pm on Jan 23, 2005 (gmt 0)

csnet - P.S Google doesnt tell us anything about how its Algo works, so why sould MSN?. If google did tell us all sites that dont rank in Google would stand a better chance!


 5:30 pm on Jan 23, 2005 (gmt 0)

MikeC is right on the bickering..as I've said MSN is what it is, lets share what we've learned and use it.


 8:48 pm on Jan 23, 2005 (gmt 0)

csnet, you're kind of missing an opportunity here to learn something. If you have a number one phrase in all three search engines, it means that each one's algo is considering you top. But each one has different ways to determine that fact. Clearly your site is not satisfying the current beta msn algo. The question is why is it not doing that? I have sites that are satisfying it. And others that are not.

I think you're assuming that your number ones in yahoo and google are the result of the same thing, but I don't think they are. For example, part of your success on google comes from the old site boost. While yahoo is based on something else. This is almost at a coincidence level. What you need to do is start working on figuring out if you can find what makes your site NOT rank, rather than complaining endlessly about that fact. A lot of my sites came out in beta ranked very high, so something made that happen. Yours hasn't. Figure it out.

This isn't a situation like the sandbox, where a penalty marker is applied to almost all new sites in a blanket way. That's a real problem, caused by several different factors, some technical, some business side, but it's stopping people from finding new material on the web, and is a symptom of deeper problems in google in my opinion.

I'd take a look at your sites, but to be honest pop culture bores me, and it's not worth the time to figure it out, although it could be educational.

It's not worth spending a huge amount of time trying to figure out the beta also because it's undergoing large modifications as they start analyzing live traffic patterns. One site I do came out at #2 for keywords, now it's number 6. That's only over a few days, so I'm not going to spend too much time worrying about it until I see the serps stabilize a bit more.

By the way, it looks like betamsn may have fixed that doorway page problem, where the main page doesn't rank and a page that just refers to the main search term indirectly does on the same site. Clearly adjustments are happening quickly.

Picking one example of a search phrase that doesn't work has little meaning. If you are looking for 'product type x', that only exists on my site in that form, you won't find it in google, but you will find it in yahoo and msn, #1 in both. If it weren't for the sandbox, it would be number one in all 3 as recent test shows. My site satisfies the algos of all 3 engines for that search term, yours doesn't.

But the reasons it's top in all 3 are different.


 10:22 pm on Jan 23, 2005 (gmt 0)

Here's how it works ... algos will be different and for that that matter will change if your site isn't at the top of the search results for any particular search phrase on an engine it's because your not ringing all the bells for that given algo metaphorically speaking....

Either try to do something about it or accept it ... there are too many people saying they should be top for this term or for that term .... like that they think they have some divine right ....


 11:44 pm on Jan 23, 2005 (gmt 0)

Some great points. Absolutely spot on.

It a nice problem to have being ranked well in Google but not MSN because you can do something about it. Where as with Google the sandbox just gets you no matter what you do.

At least here at webmasterworld these problems can be discussed.

Good luck all

[edited by: Receptional at 11:30 am (utc) on Jan. 24, 2005]
[edit reason] small language modification [/edit]


 11:16 am on Jan 24, 2005 (gmt 0)

For anyone (e.g. msndude) that thinks this search engine is ready to come out of beta try searching for counter statistics with 30 results per page. The (near) duplicate content is a wonder to behold (look for bravenet).

Now contrast those results with Yahoo and Google.

The problem seems to be that MSNSearch does not recognise subdomains as being from the same site. How long before someone tries to exploit that to blast competitors off the page.

The word "pitiful" comes to mind.



 11:37 am on Jan 24, 2005 (gmt 0)

kaled .... no better place to test than in the REAL arena .. hopefully MSN with their resources will be looking at many aspects of spam and the shortcomings of the current algo, although they are going to get bashed for putting the search out there too soon I'd say it will help them reached a satisfactory standard much quicker IMHO ...


 11:49 am on Jan 24, 2005 (gmt 0)

I'll let Kaled's specific example stand (but let's not get into the habit guys) because I think it highlights the nub of the "it's good, it's bad" argument.

As I see that example, from a user perspective, MSN has returned exactly what the searcher asked for. Lots of examples of counter statistics. True, many are from the same counter supplier, but if a user wants to compare actual STATISTICS (from entirely different websites I might add) then it would make sense to use the same data collection method - so it is easy to argue that MSN has does somenthing very clever here from a user perspective.

It is really easy if you want to choose between differnt TYPES of tracking solutions to type in a more relevent phrase. "Hit counters" or "web site trackers" seem terms that suggest the user wants to see examples of tracking systems, not examples of web statistics. So I think MSN passes the test.


 1:27 pm on Jan 24, 2005 (gmt 0)

As I see that example, from a user perspective, MSN has returned exactly what the searcher asked for. Lots of examples of counter statistics.

I used the following settings :-
Group results from the same site.
Show the first 2 results

Therefore, I would certainly NOT say the results returned were what I asked for.


PS Sorry for bending the rules on specifics.


 3:19 pm on Jan 24, 2005 (gmt 0)

to be honest to the person who's one on yahoo and google but not msn. I would be glad, and honestly wouldn't tweak my site one bit. msn won't send your site even a fraction of the traffic google and yahoo will. Why would you risk hurting your #1 ranking in the two biggest engines in order to rank in a BETA product?


 5:08 pm on Jan 24, 2005 (gmt 0)


What Tosh, i can show you loads of examples on Yahoo and Google where it lists volumes of spin off subdomains with subdomains from subdomains etc that they list entirely wrong.

MSN doesnt do that. If a subdomain has quality content in it relevent to the serch term it should list.

Why shouldnt a subdomain list if its an area dedicated to that search term with relevent content?.

Im not talking about spam, im talking about content.
Bottom line is i can show you bad examples from every search engine if i look hard enough. Frankly the other two should carry the betta tag!

MSN is doing a very good job on the overall. Its nice to see results relevent to content for once not just relevent to links!


 5:09 pm on Jan 24, 2005 (gmt 0)

MikeC, yes I am being careful about that. I am only correcting some W3C validation errors to make sure the pages are technically no problem for the MSN crawler.

I am not going for 100% W3C validation however. I use some IE6 only attributes very selectively because I like the results so I am keeping them.

Wouldn't it be a kick in the pants if after Microsoft has given us DHTML and other non-standard goodies, now MSN requires 100% W3C valdation to be included in the results, and the other SEs don't care about that. lol


 5:13 pm on Jan 24, 2005 (gmt 0)


Surfers will only leave MSN if an alternative is better. Currently both Google and Yahoo are so bad that for MSN it will be like taking candy off a baby!

No lose if you ask me. Market share for MSN can only go up!


 5:15 pm on Jan 24, 2005 (gmt 0)

>>>>Why shouldnt a subdomain list if its an area dedicated to that search term with relevent content?.

I like the way Yahoo and Tahoma handle this with links on the original SERP:

Yahoo: More results from this site
Teoma: More results from {DomainName.com}

In our case we are #1 on each original SERP. Clicking the links shows 7-10 additional links to individual pages on our site.

It helps users because the page 1 SERP is not cluttered with multiple results from the same site.


 5:31 pm on Jan 24, 2005 (gmt 0)

-- now MSN requires 100% W3C valdation to be included in the results --

wouldn't it be nice if all the webmasters did their job the right way to feed the correct HTML -- that would require a lot of work for everyone --


 6:08 pm on Jan 24, 2005 (gmt 0)

RichTC said,
Why shouldnt a subdomain list if its an area dedicated to that search term with relevent content?.

If you read my second post, you would see that my settings specifically stated that I wanted no more than 2 results from the same site and they should be grouped together.

In future, try to read posts more carefully before disagreeing otherwise people may conclude you are stupid and/or argumentative.
Of course, looking at your spelling, punctuation, grammar and typos, the former could be concluded anyway.



 12:18 am on Jan 25, 2005 (gmt 0)


sleeping with that Cat again?


 12:21 am on Jan 25, 2005 (gmt 0)

-- now MSN requires 100% W3C valdation to be included in the results --

Is this the case?

This 64 message thread spans 3 pages: 64 ( [1] 2 3 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved