Why not add a sitemap function

a quick way to expand the breadth and depth of the index

[wingslevel]

msndude - summing up some comments from this forum, lots of us like what you are doing but there seems to be some consensus that your index lacks depth.

while your bot grabs all of my small sites, it doesn't even get 10% of the big ones (50,000 pages +). not sure if this is an appetite issue (ie you only want 10% of my site in your index) or some technical barrier.

if it is a technical issue, why not set up a WebmasterWorld beta sitemap program? post a url in this forum and let us upload text files to you with urls to feed your spider - if the experiment works, you can go public with it.

[Shurik]

I'm not the Msndude but would like to comment the above post. It is not a technical issue - there is nothing inherently difficult about indexing large sites. I think MSN applies different philosophy when it comes to deep (possibly near duplicate) pages. I believe the value of the page is determined in part by its location in the site hierarchy. If its too deep it may not be too important. MSN is not trying to mimic google's index, they try to offer an alternative. And they are yet to be proven wrong...

[steveb]

Once they bill themselves as "we skim the surface and search some of the Internet", they they might not be proven wrong, but until then they are obviously wrong in that they refuse to display relevant, high quality results that it knows about. That isn't a "search engine", that is a "we aren't going to tell you" engine.

[wingslevel]

I don't think my larger sites are any more disposed to redundancy than smaller sites.

Most of the content on large sites is necessarily deep.

On my large sites, msn is indexing only the least important content.

[Shurik]

In my opinion, as i said earlier, MSN is not trying to mimic google – it would be pointless. I would argue that there is quality beyond the first 3 pages or google SERPs and if MSN could harvest those sites they would present a very attractive alternative – alternative that would allow them to start seriously competing with the market leader. If you haven’t seen an interview with Andy Edmonds and Erik Selberg I would highly recommend it.

channel9.msdn.com/Showpost.aspx?postid=128677

[wingslevel]

I think it is great to have another approach - but if that means penalizing larger sites by skimming off the 10% least valuable content, the users won't be getting the best results.

[Reno]

why not set up a WebmasterWorld beta sitemap program?

I totally agree with you wingslevel. Google's beta sitemap.xml program is -- as far as I can see -- well thought out and extremely easy to implement. In fact, I think it's the best thing Google has done in a while for webmasters, so I second your call for MSN to follow suit.

ps. I'm not sure however whether Yahoo's "urllist.txt" is particularly effective -- for me, the jury is still out on that one.