Links per Page MSN will Crawl

Links per Page MSN will Crawl since MSN does not crawl Deep

[jcmoon]

I have a website with 70,000+ webpages. The majority of these webpages are on the 3rd and 4th level. To navigate to them we have a max of approx. 100 links per page. Google has crawled almost all 70,000+ of these pages, Yahoo has crawled about 30,000, but MSN has crawled only 600.

I realize MSN is only crawling the index page and webpages at the second level. Because I have so many webpages, I was just going to put a thousand links on my index page.

Has anyone put 100's or thousands of links on their index page and seen MSN crawl them?

[RichTC]

i think you are right in your line of thoughts. MSN in effect just top skims, so i agree with your conclusion.

In answer to your question, it depends on your site structure, if your pages are dynamic forget it because msn is no longer cashing them what ever you do, it prefers static.

If your pages are static, go for it, in my experience it has the capacity to take 100, over 100 and i think it may struggle.

Its a shame if you have to resort to this to get your site indexed. It would be a good idea if MSN bot at least read a sites site map and indexed off it but its not that good currently imo.

[jd01]

I ran into a similar problem (20K pages), and this is what I came up with:

1. home page links to the main index pages of subsections.

2. home page links to main site map page.

3. the main site maps link only to the subsection site-maps

4. each subsection has it's own set of site maps linked to from the main site map.

5. each page of a subsection only links to the site map for the specific subsection it is in... not back to the main site map

IOW from a subsection you can only go 'up levels' through the main navigation, not the site maps... they are section/subsection specific.

I just tried this about a week ago, and they appear to like it, because they picked up ~250 additional pages within a few of days.

Hope this helps and makes sense.

Justin

[jcmoon]

Thanks. Regarding static vs. dynamic URLs, we use mod rewrite so that all my dynamic URLs appear static.

JD01, I will take some of your site navigation strategy and try it out. In addition which might help, we plan to use many subdomains b/c we think that MSN will view subdomains equivalent to an index page. We have seen MSN crawl the index page and next level down. We think that if we make our webpages that are one level down from the index page subdomains that MSN may view this as another index page and then go another level down (our dedicated webhosting server provides unlimited subdomains so it does not cost us anything).

What is your view on this technique?