Welcome to WebmasterWorld Guest from 54.226.25.74

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

HTML Sitemap for 10,000 Page Site

     
4:43 pm on Mar 3, 2008 (gmt 0)

Junior Member from US 

10+ Year Member

joined:Oct 31, 2005
posts:196
votes: 11


In the Google Webmaster Guidelines it says to keep the total number of links on any given page to no more than 100 links. If you have a site with 10,000+ pages, how do you construct regular human-use html sitemaps (I'm not talking about xml sitemaps here)? If you were to go by Google Webmaster Guidelines and your site had 10,000+ pages, you'd end up with 100+ site map pages, and this seems excessive and clumsy.

I was wondering what I should do in this case or what others have done who have large websites? I am trying to follow Google Webmaster Guidelines carefully so I don't want 5 sitemap pages with 1000 links each.

5:01 pm on Mar 3, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 11, 2005
posts:513
votes: 0


This is a great question, one I've often wondered about as well. I have nowhere near 10,000 pages, but I do have one site with close to 1,000, and my site map page was getting pretty large.

I split it in half recently, just to break it up a bit but I would also like to know how people with larger sites deal with providing user friendly and Google friendly site maps.

5:44 pm on Mar 3, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:1836
votes: 22


You really don't need a sitemap even if you have 1M URLs, you just need a good information structure. Google will find all URLs if PR and information structure are fine.

I'm not sure why people are still using sitemaps. If you need a sitemap for your visitors just offer a site search, I think people will not read all your sitemap links, using a site search is better in this case.

[edited by: SEOPTI at 5:48 pm (utc) on Mar. 3, 2008]

8:24 pm on Mar 3, 2008 (gmt 0)

Junior Member from US 

10+ Year Member

joined:Oct 31, 2005
posts:196
votes: 11


Ok, good advice. The reason we were using sitemaps is because it was one simple way to build links to new pages.
11:35 pm on Mar 3, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


If you want to be anal about it 10k pages is the limit of how you can easily use html sitemaps while following the 100 links per page suggestion... one sitemap page linking to you 100 sub-sitemaps, 100 links on each of the 100 sub-sitemaps equals 10,000 pages.
12:13 am on Mar 4, 2008 (gmt 0)

Junior Member from US 

10+ Year Member

joined:Oct 31, 2005
posts:196
votes: 11


steveb -

Yeah, that was the route I was going, then got tired and thought there might be a better way. thanks.

4:50 am on Mar 4, 2008 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:12214
votes: 361


Or you might try three levels...

10 x 10 x 100 = 10,000

7:48 am on Mar 9, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 20, 2002
posts:933
votes: 0


consider rolling sitemaps if your 10k page site is growing quickly.
3:04 pm on Mar 18, 2008 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:July 17, 2003
posts:701
votes: 25


You really don't need a sitemap even if you have 1M URLs, you just need a good information structure. Google will find all URLs if PR and information structure are fine.

I'm not sure why people are still using sitemaps. If you need a sitemap for your visitors just offer a site search, I think people will not read all your sitemap links, using a site search is better in this case.


Another good reason for sitemaps is it allows you to spread the PR juice a little more evenly around your site
3:20 pm on Mar 18, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 15, 2003
posts:2623
votes: 4


I have no where near 10,000 either but something I did for one of my bigger sites maybe about 800 pages was a "tree" styled view like you have on your computer with little "+" and "-" signs to click and expand directories. People like it because it is something they are used to from using a computer.

I used XML and YUI and it is clean. For 10,000 pages you could maybe organize them into groups of trees I don't know that is a lot of pages.

[developer.yahoo.com...]

[edited by: Demaestro at 3:21 pm (utc) on Mar. 18, 2008]

5:12 pm on Mar 18, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 26, 2006
posts:1397
votes: 0


Has anyone tested Sitemaps to prove it's not a complete waste of time and money?

p/g

10:50 pm on Mar 18, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 9, 2005
posts:1509
votes: 0


I would break the site up into 'sections', with one set of site maps for each main section (directory, topic).

Creating Widgets => Creating Widgets Site Map
Using Widgets => Using Widgets Site Map
Editing Widgets => Editing Widgets Site Map
Deleting Widgets => Deleting Widgets Site Map

(You can then use the above suggestion of a 'site map' of site maps, or something similar if you like. (EG linking to all site maps from the home page, depending on number of sections.))

I link only to 'section specific' site maps from within each section of the site. (Sometimes I cross link site maps to each other, but it's definitely optional.)

Example:
Home page links to Site Map of site maps (Site Map Index), which links to all above site maps.

The Creating Widgets Section of the site links only to the Creating Widgets Site Map.
The Creating Widgets Site Map links only to Creating Widgets Pages, other pages in the Creating Widget Site Map, and the Site Map Index.

This has worked to keep them a manageable size for me before, and keeps the site spiderable and surfable.

Justin

Edited, Added: Each site map link to all site map pages within a specific section of the site.

11:21 am on Mar 20, 2008 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts:298
votes: 12


You could only include new pages to speed up indexing time. However these pages are worthless without link analysis giving them rank.
12:36 pm on Mar 20, 2008 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts: 12172
votes: 61


jd01 nailed it I believe. That has been my approach from the beginning. Keep pages succinct, develop a Mother Site Map and then you have Sibling Site Maps that are strategically positioned throughout the site. Think of it like one huge book with thousands of chapters. Each chapter has its own table of contents.
12:52 pm on Mar 20, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 15, 2005
posts:380
votes: 0


If your site don't have linking structure good enough to make it indexed without a sitemap, it's worthless anyway. Spreading PR evenly through the site makes no sense too, PR is money and not every page is worth the same investment, because each page has different revenue, depending on its topic. It makes sense to make limited sitemap for top performing pages, and recently added pages, while the rest of pages has to be linked with good internal linking logic.
1:50 pm on Mar 20, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 2006
posts:569
votes: 0


with little "+" and "-" signs

funny.

I just came over a site which had a sitemap like that... but with a different URL for each setting for the 100+ expand/collapse options.

Had to give Google a try.
It knew about 200.000+ variations to that page.
Listed about 450.

Same page. Just different tabs expanded.

how come it's not dupe content? how come it's in the index? well... did I mention I was on a data gathering hunt for high trust sites? this was one

1:59 pm on Mar 20, 2008 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts: 12172
votes: 61


It knew about 200,000+ variations to that page. Listed about 450. Same page. Just different tabs expanded.

You're performing advanced searches though, correct? Google seems to know about a lot of things but, it is only going to display those things that are most relevant to the query. In advanced search queries, that is a completely different ballgame as you are seeing things that the typical searcher does not.

So, in the scenario you describe above, is it possible that Google has determined which of those 200,000 variations is the authority and will show that a majority of the time for who knows what type of searches. And then, for more long tail queries, one of those 450 results now come into play?

Sitemaps are another tool for Webmaster to utilize and control the flow of what is and isn't getting indexed. They are a great addition to the "funneling" and/or "sculpting" process. Take a site now that is having a tough time from an indexing standpoint. Build sitemaps for the pages that "should" be indexed. Drop a robots metadata element on those pages that shouldn't be indexed. Get the bot to "go with your flow". Multiple sitemaps are an integral part of any large scale website.