Welcome to WebmasterWorld Guest from

Forum Moderators: mademetop

Message Too Old, No Replies

Dynamically Created Pages and Crawlers

Dynamically Created Pages and Crawlers

5:15 am on Jan 7, 2005 (gmt 0)

New User

10+ Year Member

joined:Dec 27, 2004
votes: 0

I read that GOogle and other search engine cannot spider or crawl dynamically generated pages. That is, if a product record is in a database and it is being displayed on a page that pretty much like a template. Is this true? What's the workaround?

please advise.

5:46 am on Jan 7, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
votes: 0

By the time Google sees your code it's already in plain old HTML/XHTML. So from that perspective dynamic pages aren't a problem.

Unfortunately many dynamically generated pages also include a lot of variables in the query string. That's what the search engines have problems with.

IMO a page that's a site map with all the URLs you want the search engines to spider is the best workaround.

5:57 pm on Jan 11, 2005 (gmt 0)

New User

10+ Year Member

joined:Jan 11, 2005
votes: 0

Pages with long query strings tend not to get indexed as well static pages.


"We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index."

So, what do you do if for example you run a e-commerce shop using dynamic pages? If your fortunate enough to being using an apache server you can consider using mod_rewrite:


"This module provides a rule-based rewriting engine to rewrite requested URLs on the fly."

Or, if you cant or dont want install mod_rewrite you can create a static version of your site:


It basically converts any dynamic site into a static version for search engines.

Good luck!