DiscoStu - 6:13 pm on Aug 7, 2009 (gmt 0)

So you can have a maximum of 50 K links on an xml sitemap. But what about the html sitemap? Do the search engines use the html sitemap at all if there's an xml sitemap in place? I.e. should an html sitemap be strictly for users, or should it be used as a way to get spiders to crawl the site more thoroughly?

For smaller sites (say <100 pages) xml & html sitemaps can look the same = list everything on one page. This works both for search engines and for users. But let's say you have a site with 20 K pages. That works fine for the xml sitemap, but not an html sitemap (not user friendly - and having a html page with 20 K links on it *seems* like a bad idea in general?).

My understanding is that the xml sitemaps always should be exhaustive, covering every single URL on the site. But on very large sites ( 1 mil+), does it make sense to even try to cover every url in html sitemaps? Or should you forget about the search engines here and just make a user friendly directory of your site (and let the search engines use the exhaustive xml sitemaps to aid them in indexing, and leave the html sitemap(s) for humans only)?