Forum Moderators: Robert Charlton & goodroi
My question here is, should one worry only about Googlebot crawling a website?
What about rest of the search engines? More importantly, what about the user?
A well optimized site should not only have xml and txt sitemaps, but also a well defined HTML site map. I have seen some very good examples of user and SE friendly site maps that go to the extent of actually writing description to the links apart from listing them in a good hierarchical order.
For dynamic websites one should list the pages that are otherwise unapproachable to the SEs (where a query has to be performed) in the xml, txt and html site maps. This to my mind will improve the indexing of the site (of course one follows the general rules of not listing too many links and all).
A well-designed modern webpage, however, has many entrances. Beam us to the appropriate shelf, scotty. Google does.
A sitemap still has its value in making sure the spiders will index all pages, but there are several means to acchieve this. Many websites nowadays cover thousands if not millions of pages, and thus have grown to such a complexity, that noone expects visitors to click through severral levels of sitemaps.
Even today I find myself using them occasionally to find pages or topics I might have overlooked or missed.
However, they should not be part of the main way people get around the site. Treat them like they are the index of a book - which is your site.
the index of Kant's "critique on pure reason" is really worth taking a look at: It covers up to (if I remember correctly) seven levels of organizing thoughts.
Basically, I would agree to you, that well-ordered sitemaps still have their value in helping the webmaster organize the site and helping the visitor find out about the way the whole site is structured. But there are many websites meanwhile, where page-loading time sets strict limits to this approach.
One of the major reason, why many webmasters have abandoned html-sitemaps, is the fact, that more than 100 links on a single page are said to dilute google-pagerank too much. It is generally assumed as a thumb rule that page A linking to less than 100 pages of type B inherits a pagerank of PR(A)-1 to the B type, whereas this is lowered to PR(A)-2 if page A links to more than a 100 pages. Therefore many webmasters have shifted towards splitting their sitemap into more than one page of the A-type. A site with - lets say - 1500 pages and 15-20 part-sitemaps could ensure with such a strategy, that all of these 1500 pages get indexed with a PR(startpage)-2.
This is only a thumb-rule and many experts would probably cry "veto". But this thumbrule is quite widely spread, I think. It is a prejudice, which somehow works against googles first law: "concentrate on the user, all else will follow."
