Forum Moderators: mack
Is this happening to anyone else?
so I check a few of my competitors. One has 8000 pages on Google, 400 or so on Msn. Another site of mine (extremely white hat & not really commercial) has 90,000 on G (half are real, the rest are supplemental since the re-write) and only 1200 pages on MSN. CNN has 800,000+ on Google, about 50,000 on MSN.
Hmmm...
Another a new site of mine got indexed in google, it had about 1000 page only 11 are indexed, but the wiered thing is that the same case for MSN.com also only 11 pages indexed and its not the same pages.
The most unblieveable thing is Ask Jeeves, it crawled over 120,000 pages last month, only 115 pages indexed.
I guess, one should pay for the directory (related to the search engines, i.e. Yahoo related to venture, ...) inclusion and then get more page indexed.
Hope I was wrong
Not bad :)
So I have five sites that have the same type of category structure and I tested this theory by linking the internal pages of the 5 sites together without cross linking them.
The result was about 1,000 pages indexed per day for 2 weeks per site.
The sites all averaged 1,000 page increase every day that I checked.
But with all those pages indexed you would think my traffic would have increased on those sites? It did but I didn't notice it at all.
Maybe the pages have to get some kind of page rank before they actually rank where they should? So I'll wait a month and see if the traffic increases.
But to me they are trying to rank pages similar to how google does it except they only value single pages and not entire sites PR like google.
So each individual page has to get it's own links to rank well. And if the page has no external links pointing to it and it takes more than 2 clicks from the home page to get to that page then MSN thinks that page isn't important enough to index.
This is one of the reasons MSN's results aren't quite up to par because they don't index deep enough to get the good pages from a site.
I think they should index sites backwards. Just index the home page and then index from the deepest pages to the shallow pages because in reality the deepest pages are really the most important for most sites.
That would give them much better results than all the engines if you ask me. Just give the pages with the least links pointing at them the highest rank.
HAHAHA! Negative page rank! The next generation of SEO.
If the above is true, this algo is inteded to favor blog comment spam (and link buying to a smaller degree).
I haven't seen the 5 levels comment, but clearly they aren't doing that. Anything more than one click seems to be very likely ignored, and even one click pages can be ignored.
What a "search engine", titles of pages ignored, location of ISP and not content is the prime criteria, 100 five page sites crosslinking to each other can all be indexed and algo-loved, while one 500 page site gets 5% of its pages indexed.
Worst conceptual thinking ever.
They must give pages some kind of page rank because if they didn't they wouldn't index as deep as they do now.
One of my sites they indexed 3 levels deep without any of my help. So just the popularity of the home page was enough to get 3 levels indexed.
I am assuming they are trying to keep their index as small as possible while still serving good enough results while they test everything and learn from their engine.
Once they start to trust their engine I think they will turn up the knob and index deeper. It's easier to work on a small database than it is to work on a big one.
MSN seems to index things a bit differently than most too. I like the way they list things..
S
I just draw 5 dots in a circle. each dot = one of my websites. Then just connect the dots without cross linking sites together.
It only works one way if you have less than 5 sites. Each internal page should be able to link to 1 page on 2 other sites. But do not cross link the pages! Like A - B And B - A is bad.
Of course I haven't seen a big increase in traffic but the traffic seems to be slowly going up every day. So far it's been 3 weeks since I did this and it has definately increased my pages being indexed on all 5 sites in MSN.
I have looked at the number of pages indexed relative to Google on some different types of site. Google-optimised db-driven sites seem to do relatively badly, news/info sites are doing a lot better.
I have a theory about why MSN is working in this way. I think what they are trying to do is to scoop up authoritative information from news and information sites that are regularly updated.
So it will not matter where a story ends up (ie deep in a well-catalogued archive), what matters in terms of importance to MSN is "was this information at some point on or near the home page?"
This is an uncomfortable situation for those like myself who run large db-driven sites and have done well out of optimising them for the Google algo.
However, MSN has to distinguish itself from Google, and it can be argued that their approach has its own validity.
To test my hypothesis I've added a new page close to the home page that links to all the previous day's new data. If anyone is interested, I'll report back on how this works.
However, MSN bot has diligently put the pages in their index. Initially, I thought that they were spidering in the pages but not putting them in. But, on further research, I realized I was not using the proper commands to see what pages are in their index.
First, don't rely strictly on this command - site:www.yoursite.com
Using that command I was only able to pull up 250 results for my site (which is over 5000 pages). And much of the results were very odd indeed.
Instead, to find if your deep pages are in the MSN index, try doing the following:
site:www.yoursite.com "unique phrase for your site shared on multiple pages"
The above type of search works superbly well if you have a common phrase shared among the pages of your site. After doing that search, I realized MSN had indeed indexed my deep-down pages. It wasn't ranking them well, but guess that is another story for another day. :)
Jim
One new oddity that I've seen is that my home page is beginning to rank well for phrases that include words not on the home page - but phrases for which I have indexed pages that I'd expect to do well.
For example, the site is ranking number one (out of around 10,000) for something along the lines of red widgets [town]. The word [town] does not appear on the page, or in the code. The word red appears once, the word widget appears once. The phrase red widgets brings up around *8 million* results. On this search the home page of the site is top 30, despite the minimal on-page optimisation.
This may seem unfair or unreasonable, but in fact the site is a vast and market-leading source of widgets, red and otherwise, including some in [town], so it's actually not a bad result from a user's point of view
This raises an intriguing new possibility - that MSN search will go through a site, and when it finds a pages within a site that are highly relevant to a particular phrase, that it boosts the home page of that site for that phrase.
This is totally counter-intuitive to Google-orientated optimisers, but it makes a kind of sense - when you optimise for Google, do you REALLY want people to land on your optimised deep page, or would you actually rather that they came to your home page and saw all you had to offer? If you're like me, you regard the individual pages as a way of hooking people in, to show you have something relevant, but really you want to move them to your home or search page as soon as you can.
Think before you answer! Don't get stuck with your Google mentality - there's more than one way to skin a search cat!
All just hypothesis for now, and all of course MHO only...
Henry
Of course that is not usually what happens even though most engines try hard.
But it would be easier on the engines if they could just index the home page and then spider all the deep pages to know what keywords to rank the home page for. Then let the website owners work it out from there?
That seems like a dumb idea but who knows with microsoft?
I see your point and respect your preference. However, there are some sites, like mine, which have sophisticated search options within the site designed to let the user specify which size and type and location of widget they would like.
My search page is better at returning the right widget to a particular user than Google is - as you would expect given our specialisation.
A user looking for red widgets in Largeville will find a page from my site by putting "red widgets Largeville" into Google. Now, my site will have maybe a hundred different pages listing differently specified red widgets in Largeville, and what I want to do when that user arrives is to steer them to my search page where they can be a little more specific about the size and shape of the red widgets they are looking for, and get a better result on which they are more likely to make a transaction.
If this is the way MSN is going and (critically important) the users understand it, then I can see it working very well for some sites.
It's not what we're used to, and it may not work out. But I don't think it's dumb - just different.
H
I really don't see MSN doing this at all though because if they did they wouldn't cache the pages they index.
I think MSN is just trying to keep their database small while they fine tune it. Once they got the bugs worked out I think they will open the gates and index more pages deeper in the sites and hopefully rank them better.
Right now MSN sends most of the traffic to my home page simply because the internal pages don't rank well for any good keywords. And the reason for that is because the internal pages don't have much link popularity like the home page does.
MSN is ranking by link popularity but they are only counting link popularity for individual pages and not entire sites like google and yahoo do.
So it looks like each individual page on a site has to get it's own link popularity to be able to rank well for any kind of keyword.
Which really does make sense to me but it makes it harder to optomize an entire site for msn without getting people to link to your internal pages.
