Forum Moderators: mack
woop01, thanks for posting your story about robots.txt
In 2 years, Yahoo, Google, Teoma (Ask), and Mama never excluded a site if it did not have a robots.txt file. The assumption seemed to be no robots.txt indicated permission to crawl everything. As I have posted on here many times, several pages of our site have been the #1 results for 2 years on all those SEs on 2 -4 keyword searches, with no robots.txt file. Not a single one of our pages shows up anywhere in MSN beta. So, it sounds like you MAY be describing our problem.I just added a robots.txt to our root to see if THAT might be MSN beta's problem with listing our site. The file contains some comments and:
User-agent: *
Disallow:If someone reading this is not familiar with robots.txt syntax, there are examples and a robots.txt file checker here:
Make sure you use an editor that breaks lines Unix style (line feed w/o carriage return). The Notepad editor, like most Windows platform tools, adds a carriage return with the line feed which won't work.
Sorry for the delay on responding to this part of the thread. I just want to quickly clarify this. MSNBot will support both and either. That is, you can use a carriage return and line feed or just one of those.
Additionally, like other crawlers we will crawl and index a site that has no robots.txt.
- msndude (msd)
What's more when a specific search is entered it is usually the top level index page that is shown as opposed to the more relevant second level page even though it is in the database... why?
Because this is still a beta (I hope).
I am experiencing the same thing, the search results are pointing to a weaker page or a page that points to the correct page.
I am asking those who have posted here on this thread saying that MSN results are excellent to ....
I'll answer that. They are using the same criteria as you set forth - only the opposite. Your site(s) not included or ranking so they are censoring and crooked. Their sites are included and ranking well so it's the best SE out there.
Problem is that criteria is flawed big-time. You don't base/judge SE results on the criteria of your site. You also can not judge a SE by one search term here and there. You would have to use a large number of terms, say the top 10,000 terms searched for in a day and then run through all of the top SE's and have independent analysis. You also cannot tell the persons intent performing the search on many phrases or terms and neither can a SE.
Sorry, but I think they are doing OK for how long they have been indexing. There have been quite a few SE's over the years that never had results as good as what's showing now (and they had years to perfect them).
I think you are just miffed because you have a site ranking well in Google but not MSN. Frankly its time to stop moaning and time to look at ways to improve your content and website so that you do feature well in MSN also. MSN is NOT Google - thank God!
Also, at least with MSN you have a chance to improve you MSN SEO of your site, rather than in Google where many, many websites dont get that chance due to the sandbox! Think yourself lucky!
fleapit - you assume wrong that only top level is indexed. I have a number of pages that list well. Of these pages what i would say is that they are rich in content, are updated on a regular basis and have correct titles and quality links etc. Content is King on MSN.
How many times do I have to to post that our site has more current and historical content about this actress than any other site on the Internet?
Our site is not just the #1 site for this actress 2 word keyword search on Google, but also on Yahoo, Teoma, Ask, Mamma. I have not mentioned it before but it is also #1 on Alexa, Alta Vista, and Excite. I went to a 3rd party site that has links to over 20 search engines. We are always #1. ONLY MSN beta excludes our site and dozens of other sites done by others in this genre.
Because MSN does not disclose how it rates sites, none of us can do anything except speculate about what causes a site to be included or excluded in MSN SERPs unles MSNdude posts something to set us straight about a particular item. :-)
As I mentioned, we provide links to 3rd party sites that we believe would interest our visitors. For all I know, one of these sites may link to a site that links to a site that does not meet MSN's criteria, so every site that links to that site even indirectly is dropped from MSN SERPs. But this is speculation because MSN is not disclosing it's methods. We certainly meet or exceed all the suggested criteria published in the MSN beta help pages.
As in the buy oranges in Korea example provided by roycerus above, MSN search logic must be seriously flawed to return such riduculous SERPs.
We could try dropping all links to 3rd party sites and hope that MSN will include us in the SERPs after that, but our site would be less useful to our visitors. In addition, the reasonable use of link exchanges make the Internet more useful. Can you imagine an Internet where the ONLY way to find a new site is through a SE? Perish the thought! lol
Then they would have good search results every time if we knew how to optomize for them.
If everyone knew the "secret" then it would just come down to link popularity and sites would be ranked accordingly.
Link popularity should be based on the amount of links pointing to a page and not Page Rank which is easy to manipulate by buying links from high PR sites. (Or was easy?).
hmmmmmm
There are loads of example of good and bad serps in msn, but so what. If it's generally good people will use it, if not they won't. It will never be perfect and neither will any of the others.
I imagine there are very few sites that are no. 1 on the three main se. Anyone that's getting 2 out of 3 is doing pretty well in my opinion.
I don't think you can try and lump this disaster in with anything else or even try to compare it with any REAL search engine, this is the worst case of "giganticstuffupitis" I've ever seen.
Google or Yahoo have NEVER been this bad , in fact I can't think of any SE that has been this embarassingly awfull.
Heads will roll at MSN.
In the case of Google almost every search i do these days has at least 3/4 directory sites in the top ten or sites returned that have bugg@r all content.
What i would say is that without any doubt google favours older sites over new - Where as i believe that MSN treats all sites as equal from the starting point and then looks at content Oh and MSN has no sandbox (yet!) .
Csnet - What you say is interesting but i dont want to sound cocky but i reccon if i looked at your actress site i would be able to quickly see why you are not listing in MSN. If you have the content and your pages are titled correctly and you have quality links to the actress name, your site should be listing well for those key words but i guess it depends on what keywords you are trying to list under in relation to the content of the page in question. I find that with Google once a site ranks well due to anchor links they list it for almost anything, i see this often for old sites.
I have a site with a section dedicated to "insurance Jobs" it has links to it with that Anchor text, its very rich in content and has over 10,000 Insurance jobs in it - so i would say that should be relevant to that search term in google. The section is 9 months old. Currently in Google it lists after sites relating to Insurance Quotes and buy Life insurance for an "Insurance Jobs" search term, so i could spend all day telling you how cr@p google is about bringing back results relevant to search terms couldnt i?.
Bottom line is that, as i see it Google is still based on the principle that Links are king V MSN which is clearly about content first.
Two different engines with different algos. It is possible to crack all three search engines but i havent done it myself yet, but a few others have
I think you're assuming that your number ones in yahoo and google are the result of the same thing, but I don't think they are. For example, part of your success on google comes from the old site boost. While yahoo is based on something else. This is almost at a coincidence level. What you need to do is start working on figuring out if you can find what makes your site NOT rank, rather than complaining endlessly about that fact. A lot of my sites came out in beta ranked very high, so something made that happen. Yours hasn't. Figure it out.
This isn't a situation like the sandbox, where a penalty marker is applied to almost all new sites in a blanket way. That's a real problem, caused by several different factors, some technical, some business side, but it's stopping people from finding new material on the web, and is a symptom of deeper problems in google in my opinion.
I'd take a look at your sites, but to be honest pop culture bores me, and it's not worth the time to figure it out, although it could be educational.
It's not worth spending a huge amount of time trying to figure out the beta also because it's undergoing large modifications as they start analyzing live traffic patterns. One site I do came out at #2 for keywords, now it's number 6. That's only over a few days, so I'm not going to spend too much time worrying about it until I see the serps stabilize a bit more.
By the way, it looks like betamsn may have fixed that doorway page problem, where the main page doesn't rank and a page that just refers to the main search term indirectly does on the same site. Clearly adjustments are happening quickly.
Picking one example of a search phrase that doesn't work has little meaning. If you are looking for 'product type x', that only exists on my site in that form, you won't find it in google, but you will find it in yahoo and msn, #1 in both. If it weren't for the sandbox, it would be number one in all 3 as recent test shows. My site satisfies the algos of all 3 engines for that search term, yours doesn't.
But the reasons it's top in all 3 are different.
Either try to do something about it or accept it ... there are too many people saying they should be top for this term or for that term .... like that they think they have some divine right ....
It a nice problem to have being ranked well in Google but not MSN because you can do something about it. Where as with Google the sandbox just gets you no matter what you do.
At least here at webmasterworld these problems can be discussed.
Good luck all
[edited by: Receptional at 11:30 am (utc) on Jan. 24, 2005]
[edit reason] small language modification [/edit]
Now contrast those results with Yahoo and Google.
The problem seems to be that MSNSearch does not recognise subdomains as being from the same site. How long before someone tries to exploit that to blast competitors off the page.
The word "pitiful" comes to mind.
Kaled.
As I see that example, from a user perspective, MSN has returned exactly what the searcher asked for. Lots of examples of counter statistics. True, many are from the same counter supplier, but if a user wants to compare actual STATISTICS (from entirely different websites I might add) then it would make sense to use the same data collection method - so it is easy to argue that MSN has does somenthing very clever here from a user perspective.
It is really easy if you want to choose between differnt TYPES of tracking solutions to type in a more relevent phrase. "Hit counters" or "web site trackers" seem terms that suggest the user wants to see examples of tracking systems, not examples of web statistics. So I think MSN passes the test.
As I see that example, from a user perspective, MSN has returned exactly what the searcher asked for. Lots of examples of counter statistics.
I used the following settings :-
Group results from the same site.
Show the first 2 results
Therefore, I would certainly NOT say the results returned were what I asked for.
Kaled.
PS Sorry for bending the rules on specifics.
What Tosh, i can show you loads of examples on Yahoo and Google where it lists volumes of spin off subdomains with subdomains from subdomains etc that they list entirely wrong.
MSN doesnt do that. If a subdomain has quality content in it relevent to the serch term it should list.
Why shouldnt a subdomain list if its an area dedicated to that search term with relevent content?.
Im not talking about spam, im talking about content.
Bottom line is i can show you bad examples from every search engine if i look hard enough. Frankly the other two should carry the betta tag!
MSN is doing a very good job on the overall. Its nice to see results relevent to content for once not just relevent to links!
I am not going for 100% W3C validation however. I use some IE6 only attributes very selectively because I like the results so I am keeping them.
Wouldn't it be a kick in the pants if after Microsoft has given us DHTML and other non-standard goodies, now MSN requires 100% W3C valdation to be included in the results, and the other SEs don't care about that. lol
Surfers will only leave MSN if an alternative is better. Currently both Google and Yahoo are so bad that for MSN it will be like taking candy off a baby!
No lose if you ask me. Market share for MSN can only go up!
I like the way Yahoo and Tahoma handle this with links on the original SERP:
Yahoo: More results from this site
Teoma: More results from {DomainName.com}
In our case we are #1 on each original SERP. Clicking the links shows 7-10 additional links to individual pages on our site.
It helps users because the page 1 SERP is not cluttered with multiple results from the same site.
wouldn't it be nice if all the webmasters did their job the right way to feed the correct HTML -- that would require a lot of work for everyone --
Why shouldnt a subdomain list if its an area dedicated to that search term with relevent content?.
If you read my second post, you would see that my settings specifically stated that I wanted no more than 2 results from the same site and they should be grouped together.
In future, try to read posts more carefully before disagreeing otherwise people may conclude you are stupid and/or argumentative.
Of course, looking at your spelling, punctuation, grammar and typos, the former could be concluded anyway.
Kaled.
sleeping with that Cat again?
Is this the case?
