Why won't Google index my dynamic pages?

[ichthyous]

I have an online store which is quite extensive at this point. For some reason Google just will not index these dynamic pages. All of my HTML pages get spidered often and are included in Google's index. Google currently has about 200 dynamic pages indexed, this is only a small part of the total, and these pages don't show any totle or description...just te URL! Yahoo has indexed many more pages from the store with proper titles and descriptions with no problem. Following the advice of others in this forum, I built a sitemap for the store and Google does spider the sitemap, but won't follow the links. I have tried to keep the pages to no more than 2 to 3 levels deep, and the pages themselves are not very large (around 65k with graphics)...I have checked my robots.txt file and there's no problem there. Can anyone think of a reason for this behavior? The store has a 0 page rank, would this cause Google to avoid it? Thanks!

[troels nybo nielsen]

> The store has a 0 page rank, would this cause Google to avoid it?

A good rule of thumb is that the higher PageRank you have, the more pages Google will crawl.

[storevalley]

I built a sitemap for the store and Google does spider the sitemap, but won't follow the links

Try using static URLs that map to your dynamic pages ... this seems to improve the speed at which pages are indexed by Google.

[ichthyous]

Maybe i should clarify...I am using a cloaking system (I am also running a Flash site) so the cloaked index page has a PR of 3, but that doesn't help the real dynamic pages to get indexed. The sitemap page are all static HTML pages with links to the dynamic pages. I used Xenu to automatically generate the site map as there are thousands of pages. Other than sitting and creating one cloaked page for every dynamic page I can't see what else to do.

[ichthyous]

Also...of the 200 or so pages that are indexed by Google only a few of them have a title or description. The rest are just url strings. I have all my meta data in place, so that's not the problem. Yahoo has no problem indexing the dynamic pages and they appear with proper title and description. As a result my Yahoo traffic is starting to approach my Google traffic.

[storevalley]

I used Xenu to automatically generate the site map as there are thousands of pages

They are all on one page? It is often quoted that Google only reads the first 101k of a page. Perhaps this is the problem?

Perhaps Google's cache of your site map will offer some clues ...

[ichthyous]

Oh...I kept this in mind and made sure that no single page was over 100k. Since my dynmaic pages number 8,500 at this point the sitemap is 50 HTML pages long!

[yaelede]

Keep the links on a given page to a reasonable number (fewer than 100).

This is quoted from the "Google guidelines for webmasters"

If you have 8500 pages diveded into 50 html link pages, then it means that every page contains 170 links...

[Marcia]

>>only a small part of the total, and these pages don't show any totle or description...just te URL!

Do those show up as in the supplemental index? Or only when click for additional pages from the site?

>8,500

And how much, what percentage of the content on each of these pages is unique, differentiating them from all the others?

[ichthyous]

They show up in the regular results, mixed in randomly with the rest of my indexed pages. A few show the proper title, but most are just a URL. The amount of these pages in the index deems to go up and down daily, but only by about 20 pages or so. The great majority of the content is duplicates...there are only about 2,500 items, but my store technology doesn't allow me to enter the product once and refer to it in several categories...so i am forced to enter duplicate items under several categories. Even if Google indexed only the unique pages I would be in good shape. I recently altered all of the department, category and subcategory titles in an effort to give the SE's a bit more to grab on to as well. Thanks for the help guys (and girls)

[Swanson]

I have seen this in many e-commerce sites, it seems that G thinks there is duplicate content because of the querystrings.

If at any point an additional parameter is added to the querystring then google suspects that it can't properly index the document and that it is a duplicate - and yes it is technically and many ecom packages do this.

In ecom often "add to basket" or anything similar can add a param to the page and therefore flag it as a duplicate. In this case G won't crawl the site.

If you get URL only listings create a set of product/category pages that are product listings (hard code categories etc.) and link them from the product page.

[ichthyous]

At first I thought it was the query string length but some short query string products show up with a title and description while others don't. I didn't undertsand the last part of what you wrote:

"If you get URL only listings create a set of product/category pages that are product listings (hard code categories etc.) and link them from the product page."