Google friendly site map - 20,000 + Products

How do I get them all indexed

[instabill]

I have been sucking up all the wonderful information on this site for about a month now, but I am having trouble determining how to build a site map for a site that sells 20,000+ versions of their widget. Each widget has it's own dynamically built webpage. Google doesn't seem to have trouble with the dynamic generated pages but currently has only indexed about 10% of the products.

1. How do I build a site map to help Google easily suck up all the products while still maintaining the 100 link limit that seems to be the rule?

2. Are there other ways to help Google and customers find all of the widgets in the site?

Bill

[edneil]

What is the 100 link limmit?

[excell]

I would build a directory of products via category and sub category so that folks can search for the stuff or browse the directory.

There are a number of products available to enable this that can produce the static HTML pages off your database quite easily.. This would benefit the user and the search engines in indexing your site.

[excell]

It's not a rule edneil
it's a comfortable size for a list of links before you start choking the robots!

[instabill]

excell,

I have an extensive category tree that is dynamically built as well. Do I need to convert this as you suggest to static HTML pages?

Can you help me out with where to find the products you mentioned. Sticky me if you have too.

Thanks,

Bill

[Go60Guy]

I agree entirely that building a directory will be very beneficial to your users.

Just recently, I've done this on a smaller scale. However, I found that there may be a haitus in which Google has not fully spidered the new sitemap subpages (eleven of them linked to from the main directory page). So, those pages currently have a greyed out PR. This has downgraded loads of other pages, it seems, to a PR1. Consequently, traffic is down due to lower rankings on a slew of pages.

I believe this situation will sort itself out in time, hopefully, by February.

The benefit is mainly to users and it has rendered the sitemap pages in keeping with Google's suggestion to limit links to about 100 per page.

[instabill]

Are there any places I can go to find information about how I should build this directory?

Bill

[excell]

OK.. the directory itself should be acting as a "sitemap" there must be a post around here that speaks of how the dynamically generated pages creates the static HTML? For browsing?

I don't attend to the technical details of how that happens myself so I cannot personally answer you, but I am sure others here can point you in the right direction.

As mentioned this could be a matter of time, I however would be looking into your set up to ensure that it will.

Google wants to see what you have in your database and it is just a matter of getting it set up to allow access in the easiest way.