HTML Sitemap for 10,000 Page Site

[aok88]

In the Google Webmaster Guidelines it says to keep the total number of links on any given page to no more than 100 links. If you have a site with 10,000+ pages, how do you construct regular human-use html sitemaps (I'm not talking about xml sitemaps here)? If you were to go by Google Webmaster Guidelines and your site had 10,000+ pages, you'd end up with 100+ site map pages, and this seems excessive and clumsy.

I was wondering what I should do in this case or what others have done who have large websites? I am trying to follow Google Webmaster Guidelines carefully so I don't want 5 sitemap pages with 1000 links each.

[AndyA]

This is a great question, one I've often wondered about as well. I have nowhere near 10,000 pages, but I do have one site with close to 1,000, and my site map page was getting pretty large.

I split it in half recently, just to break it up a bit but I would also like to know how people with larger sites deal with providing user friendly and Google friendly site maps.

[SEOPTI]

You really don't need a sitemap even if you have 1M URLs, you just need a good information structure. Google will find all URLs if PR and information structure are fine.

I'm not sure why people are still using sitemaps. If you need a sitemap for your visitors just offer a site search, I think people will not read all your sitemap links, using a site search is better in this case.

[edited by: SEOPTI at 5:48 pm (utc) on Mar. 3, 2008]

[aok88]

Ok, good advice. The reason we were using sitemaps is because it was one simple way to build links to new pages.

[steveb]

If you want to be anal about it 10k pages is the limit of how you can easily use html sitemaps while following the 100 links per page suggestion... one sitemap page linking to you 100 sub-sitemaps, 100 links on each of the 100 sub-sitemaps equals 10,000 pages.

[aok88]

steveb -

Yeah, that was the route I was going, then got tired and thought there might be a better way. thanks.

[Robert Charlton]

Or you might try three levels...

10 x 10 x 100 = 10,000

[nuevojefe]

consider rolling sitemaps if your 10k page site is growing quickly.

[surfgatinho]

You really don't need a sitemap even if you have 1M URLs, you just need a good information structure. Google will find all URLs if PR and information structure are fine.

I'm not sure why people are still using sitemaps. If you need a sitemap for your visitors just offer a site search, I think people will not read all your sitemap links, using a site search is better in this case.

Another good reason for sitemaps is it allows you to spread the PR juice a little more evenly around your site

[Demaestro]

I have no where near 10,000 either but something I did for one of my bigger sites maybe about 800 pages was a "tree" styled view like you have on your computer with little "+" and "-" signs to click and expand directories. People like it because it is something they are used to from using a computer.

I used XML and YUI and it is clean. For 10,000 pages you could maybe organize them into groups of trees I don't know that is a lot of pages.

[developer.yahoo.com...]

[edited by: Demaestro at 3:21 pm (utc) on Mar. 18, 2008]

[potentialgeek]

Has anyone tested Sitemaps to prove it's not a complete waste of time and money?

p/g

[jd01]

I would break the site up into 'sections', with one set of site maps for each main section (directory, topic).

Creating Widgets => Creating Widgets Site Map
Using Widgets => Using Widgets Site Map
Editing Widgets => Editing Widgets Site Map
Deleting Widgets => Deleting Widgets Site Map

(You can then use the above suggestion of a 'site map' of site maps, or something similar if you like. (EG linking to all site maps from the home page, depending on number of sections.))

I link only to 'section specific' site maps from within each section of the site. (Sometimes I cross link site maps to each other, but it's definitely optional.)

Example:
Home page links to Site Map of site maps (Site Map Index), which links to all above site maps.

The Creating Widgets Section of the site links only to the Creating Widgets Site Map.
The Creating Widgets Site Map links only to Creating Widgets Pages, other pages in the Creating Widget Site Map, and the Site Map Index.

This has worked to keep them a manageable size for me before, and keeps the site spiderable and surfable.

Justin

Edited, Added: Each site map link to all site map pages within a specific section of the site.

[Johan007]

You could only include new pages to speed up indexing time. However these pages are worthless without link analysis giving them rank.

[pageoneresults]

jd01 nailed it I believe. That has been my approach from the beginning. Keep pages succinct, develop a Mother Site Map and then you have Sibling Site Maps that are strategically positioned throughout the site. Think of it like one huge book with thousands of chapters. Each chapter has its own table of contents.

[Wizard]

If your site don't have linking structure good enough to make it indexed without a sitemap, it's worthless anyway. Spreading PR evenly through the site makes no sense too, PR is money and not every page is worth the same investment, because each page has different revenue, depending on its topic. It makes sense to make limited sitemap for top performing pages, and recently added pages, while the rest of pages has to be linked with good internal linking logic.

[Miamacs]

with little "+" and "-" signs

funny.

I just came over a site which had a sitemap like that... but with a different URL for each setting for the 100+ expand/collapse options.

Had to give Google a try.
It knew about 200.000+ variations to that page.
Listed about 450.

Same page. Just different tabs expanded.

how come it's not dupe content? how come it's in the index? well... did I mention I was on a data gathering hunt for high trust sites? this was one

[pageoneresults]

It knew about 200,000+ variations to that page. Listed about 450. Same page. Just different tabs expanded.

You're performing advanced searches though, correct? Google seems to know about a lot of things but, it is only going to display those things that are most relevant to the query. In advanced search queries, that is a completely different ballgame as you are seeing things that the typical searcher does not.

So, in the scenario you describe above, is it possible that Google has determined which of those 200,000 variations is the authority and will show that a majority of the time for who knows what type of searches. And then, for more long tail queries, one of those 450 results now come into play?

Sitemaps are another tool for Webmaster to utilize and control the flow of what is and isn't getting indexed. They are a great addition to the "funneling" and/or "sculpting" process. Take a site now that is having a tough time from an indexing standpoint. Build sitemaps for the pages that "should" be indexed. Drop a robots metadata element on those pages that shouldn't be indexed. Get the bot to "go with your flow". Multiple sitemaps are an integral part of any large scale website.