MSN beta not going deep on large sites

They crawl, but the crawling doesn't get indexed

[Scarecrow]

After 18 months of aggressive crawling, MSN is not indexing very deep on large sites, judging from their beta.search.msn.com

This has been verified on four different large sites. A large site is one where Google or Yahoo report over 50,000 pages using the site:www.?.com command.

The counts in Google and Yahoo for large numbers are incredibly inflated using this command. They can easily be double the actual number of total pages on that site. On my own large site, the numbers are four times higher than the total number of pages on the site.

Therefore, you have to confirm your counts by adding another word to the command, as in site:www.?.com extraword

Choose a word that will bring the count down under 1,000. Counts under 1,000 are still inaccurate, but can be improved with this technique:

1) Set your preferences to 100 per page.
2) Click on the last page.
3) Scroll to the bottom. If there is a message that you can repeat the search with the omitted results included, then click on it.
4) Click on the last page and get the actual count of displayed links.

Here are the counts for site:www.webmasterworld.com:

Google: 224,000
Yahoo: 288,000
beta.search.msn.com: 10,105

And now with the extra word to bring the count under 1,000, here are the counts for site:www.webmasterworld.com everyman:

Google: 408
Yahoo: 224
beta.search.msn.com: 30

After studying why the MSN counts are so low on large sites, it appears that the more your extra word tends to appear on deeper pages, the lower MSN's coverage of that site for that extra word.

In other words, on large sites the MSN crawler has been going deep for 18 months, but the beta.search.msn.com engine is not displaying deep pages on these sites.

Unless Microsoft gets it together, their search engine will not be competitive.

[minnapple]

Users are not geeks like us.
MSN search will hold the same market share they do today if they hold the ground they have today.
What really matters is how in the future they bundle search with other MS products.

[whoisgregg]

MSN also reports different numbers for site: searches every fifteen minutes or so even if the site doesn't change... so the number has to be suspect. :(

I've done searches over the period of an hour and watched the number fluctuate between 9,000 and 30 for the same "site:mydomain.com" search (without any other terms). The site only has about 500 pages, so that's a pretty amazing variance.

[MLHmptn]

One word......BETA! HELLO! And realistically I don't know what your all talking about. The amount of pages indexed from my sites far exceeds Google's but whether that is due to Google's SH**TY indexing because some of my sites are new I dunno. My older sites are only a few hundred pages off or less from what google reports and that is only because they indexed my cgi-bin links when my robots.txt specifically tells them not too! It would be nice to see some of you people jump off your Google bandwagon! Google has been producing poor results for many, many of months now unless of course your site is old and has thousands of backlinks. Furthermore this is MSN search...It is not Google and MSN never said it was! I don't know about your industry but in my industry MSN at least shows all top 15 results being relevant, where as google shows maybe 2-3 of the top 10 that are actually relevant pages. Keep on your bandwagon, keep feeding the adwords, adsense and google's IPO. And when you invest your money in Google because it is so great don't sell of your shares because you thoroughly backed Google and it's BS!

Just a matter of opinion here....Don't take this offensive! But I am sick of the comparisons, hearing about Google this, Google that, etc... MSN is not Google and hopefully before long end users and not us webmasters start turning the tide on Google due to it's poor relevancy!

MLHmptn

[Hugene]

wow, take it easy MLHmptn (still unhappy cause you couldn't get the msn search code on your apge right ;)

we have to be productive here. we need to figure this one out as fast as possible, cause in a few months it will be live.

I say a good analysis approch for now is: everyone concentrate in their respective industries/fields, that we know well, then compare how your site is doing (you know its characteristics, content, links, etc) with respect to your competitors (who you migth know about a bit)

My only problem with my own approach is that, I don't believe the current results MSN is giving will remain. Maybe I am too optimistic, maybe I am blind and refusing to see the truth, but I just don't believe these results will remain, as I find them very poor.

So the real deal will be to apply my comparison approch, some basic testing ONCE the results have stabilized and MSN is close to a release version.

The question is, will MSN warn us a few weeks / months in advance before releasing to the public?