Spiders won't index my dynamic pages!

[ichthyous]

Hi there, I have an online store which acts as a kind of online archive of my images (I'm a photographer) and the program comes with a meta tag inclusion option where you fill in the meta tags along with the product and the code is dynamically generated. I have only seen the top level pages (e.g. the navbar links) spidered and never the dynamic pages and I have my doubts at this point that it will ever be spidered. Here is a blurb from the company concerning how the program is built: "Unlike other shopping cart systems, ShoppingQ is written in C++ and compiled into machine language for the target server platform." It would be a tremendous help if these pages could be spidered and would save me the time of having to create a separate keyword page per image. Is there any way to get the spiders to crawl these dynamic pages at all? Thanks

[BlueSky]

This type of shopping cart is a double edge sword. Compiled code usually runs faster, but the customer can't customize the source.

I checked some of the stores on ShoppingQ's site. If your photos are listed under departments or categories, the URLs for these are passing three variables. When SEs see cgi?, they are kinda reluctant to index too deeply because they don't want to crash your server or have their bot getting lost. Usually, Google will index two variables okay but gets real shy on three or more. Since you can't mod the source to get around this, one thing you could try to encourage indexing is to create a site map containing the urls of your product pages which only have two variables.

[ichthyous]

Thanks for the reply! So they actually are telling a half-truth when they advertise their store as being searchable. My product pages can be as deep as 4 levels down (Dept/Category/SubCat/Product.) The site map for this store would be MASSIVE as I have close to 1000 product pages a this point. The spiders don't need to crawl them all just a portion of them as a lot of it is the same product residing in two categories. I will try to do what you say and see if it works..thanks!

[DaveAtIFG]

Xenu is a free link checker that will optionally build a site map. It may provide a good starting point for your map, and some insight into how a spider views your site.

And in case nobody has mentioned it, welcome to WebmasterWorld! :)

[pleeker]

So they actually are telling a half-truth when they advertise their store as being searchable.

Not necessarily, but you may have misunderstood what they mean by that.

If a customer can visit your site, locate a search form, and type in a keyword to search, or maybe select from a pulldown menu, and then click a button to execute a search of your items, that's a searchable store.

But a SE spider just sees code. A spider can't enter anything in a search field and hit a button; it can't choose a category from your pull-down menu button and view that category.

The easiest way, IMO, to get dynamic content spidered is to create static links to that content. Google (and the other SEs) are willing to spider 'dynamic' content if you're willing to create a static link to it. So take the URL for each of your categories and make some text links on your home page to each category. Then give the spiders some time and they'll take care of indexing those category pages.

Hope this helps.

[ichthyous]

Actually I was very specific about SEs being able to search the store when i inquired at the company...they even have searchable meta tag inputs in their control panel so they know how important it is to people. they also know that if they didn't do that that people wouldn't buy their product. I am already working on those static links! The prospect of having all of those pages finally spidered has me all worked up...do you think that all these internal dynamic pages will give me a boost in page rank? Thanks!

[pleeker]

do you think that all these internal dynamic pages will give me a boost in page rank?

We're getting away from "Website Technology Issues" here, so you may want to head over to the Google News [webmasterworld.com] forum and ask about PageRank in there.

(To answer your question, no. PageRank is a measure of the amount of incoming links from other sites to yours. Has nothing to do with having dynamic content spidered.)

[BlueSky]

So take the URL for each of your categories and make some text links on your home page to each category. Then give the spiders some time and they'll take care of indexing those category pages.

Some of the stores I looked at were set up this way, and they aren't deeply indexed either. IMO he needs to get the bot past the three variable road block pages down to where the products are. Then he should be a-okay.

[ichthyous]

Pleeker,

Page rank also consists of rank passed on from a site's internal pages as well, that's why all pages should be cross linked. Since each of these product pages have links back to other pages on my site I am hoping that I will get a boost from the sheer number of pages cross linking.