150k pages indexed with G, less than 250 with MSN.Why?

Can't figure out why MSN does not index our site

[Dan92SLC]

How is it that we have 150,000 plus pages indexed in Google, 50,000 in Yahoo, but only 200 or so pages indexed in MSN? Organically, we average 7,000 unique visitors per week with Google, 3000 or so with Yahoo, and about 100 with MSN.

I know others are having a similarly frustrating experience. That is, you have a stable, established site with proper navigation, plenty of backlinks, and decent organic presence with the other search engines but you are getting nowhere with MSN.

What recourse do I have? I am torn between really focusing on this issue, and just laughing it off due to MSN's minute share of the search volume pie in general.

[Praxus]

Its hard to say anything without looking at the site. I'm curious as to what kind of content is on 150k pages thats not auto-generated, in which case the spam filter may be working as intended. As a webmaster, its hard to think of many sites with hundreds of thousands of pages that actually have good content.

[RichTC]

Praxus,

Its that kind of comment that endorses why msn think they are right to surface scrape rather than deep crawl a site. You can have a quality site in some sectors that may well carry a good couple of hundred thousand pages that are not autogenerated or spam as you imply. Our team have worked on many such sites.

Also, msns quality problem is not imo down to spam or autogenerated material its down to having far to much Junk ranking in its serps that it has no technology to detect hence why you see thin content junk sites with nothing more than the keyword in their domain ranking one in the serps for almost every keyword search you do on the search engine.

Dan,

You are probably right to laugh it off, they have such a small reach its not worth worrying about however, traffic is traffic and all site visitors count.

Until they start deep indexing a site you have no chance of seeing your site pages rank. We work on one superb site that has over 150,000 pages, has about 40,000 backlinks including .gov, .ac., related blue chip sites etc and is a google PR7 site. MSN have about 700 pages indexed and have it down for about 4,000 back links and it ranks for Jack all - because they cant deep crawl they cant allocate more content to it nor can they find all of the backlinks the site has gained over the last few years either hence it has no position in msn.

Meanwhile, a small thin content site we work on of about 300 pages, ranks for virtually every page on its site. Its a google PR4 site, low quality has about 700 backlinks of which msn has found about 100. - Its a poor site by contrast to the one above but msn love it!

In conclusion, i would laugh it off as i fear you may be waiting a long time for msn search to provide quailty and a change of name to live.com doesnt make one dot of differance. A pig is a pig even if its called babe!

[vite_rts]

A site with 150,000 pages with 5 team members , editing 40 pages a day each, would take,,,,

750 days

or

2.05 years working 7 days a week, no holidays, weekends, etc

or

3 years in the real world, having done that, how do they review it continously?

How believable is that?

[Dan92SLC]

we are a quality site. I am the e commerce manager here at a major catalog/direct marketing (somewhat niche) business. We have had a functional online store since 97, and currently use the same merchandising/site search/product deployment/display as some very very well known web retailers. Does our site actually have 150k pages? no. However, we do sell over 25,000 products. I have my opinion as to why goog has inflated page count, but that is not germane to this discussion.

I am specifically trying to figure out why msn is not indexing our site the way all other major search engines do.

To answer how 'believable' is that...let me tell you that we spend hundreds of thousands of dollars on PPC advertising, that we have 2 in house programmers, a 10 person call center, and a 4 person purchasing dept. Our goal for the next year is to launch another 5,000 products. We do 8 figures/year in sales...so you are barking up the wrong tree if you think we are doing anything shady...

[edited by: Dan92SLC at 12:33 pm (utc) on Sep. 26, 2006]

[Praxus]

Wasn't thinking you're doing anything shady; just mentioned the content
point because it often is relevant with huge sites. There've been guys here complaining that their 5 million page site isn't fully indexed by MSN, then they disappear when content is brought up.

A big shopping site is a great example of a site that legimately would have many thousands of pages. From my experience, MSN simply just doesn't index as many pages for any large site as the other engines.

[jd01]

Ditto...

I run an online directory of about 14K pages.

Nevermind the deep links from places like the U of Kentucky, Florida, Maryland, etc. I've got 220 pages, most of which are URL only.

I recently did a large update to my directory software/navigation and have seen quite a bit more spider activity from MSN. Maybe they need extra navigation to get them into a deeper crawl?

(I'm running about 600 pages of sitemaps just to try and get MSN Bot crawling a little.)

Justin

[Frederic1]

From my personal experience, MSN seems to have problems with hyphenated urls. They are indexed but very badly. I just replaced hyphens ("-") by commas (",") and much more pages got indexed! There seems to be some kind of filter to prevent too many pages from being indexed (since url rewrited sites often use hyphens as a separator).