Dynamically Created Pages and Crawlers

[RChemtob]

Hi,
I read that GOogle and other search engine cannot spider or crawl dynamically generated pages. That is, if a product record is in a database and it is being displayed on a page that pretty much like a template. Is this true? What's the workaround?

please advise.
rafael

[GaryK]

By the time Google sees your code it's already in plain old HTML/XHTML. So from that perspective dynamic pages aren't a problem.

Unfortunately many dynamically generated pages also include a lot of variables in the query string. That's what the search engines have problems with.

IMO a page that's a site map with all the URLs you want the search engines to spider is the best workaround.

[justin_s]

Pages with long query strings tend not to get indexed as well static pages.

[google.com...]

"We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index."

So, what do you do if for example you run a e-commerce shop using dynamic pages? If your fortunate enough to being using an apache server you can consider using mod_rewrite:

[httpd.apache.org...]

"This module provides a rule-based rewriting engine to rewrite requested URLs on the fly."

Or, if you cant or dont want install mod_rewrite you can create a static version of your site:

[staticshell.com...]

It basically converts any dynamic site into a static version for search engines.

Good luck!