homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Dynamic pages
Will they get spidered and listed

 10:10 pm on Feb 11, 2003 (gmt 0)

Will Google see my dynamic pages?

Does Google ingnore the link below because it has a?

What is the final or current say on dynamic pages and gettting them spidered and listed?

I thought Google did not follow or list dynamic pages?




 10:13 pm on Feb 11, 2003 (gmt 0)

Hi There,

If you keep your querystrings to a maximum of two short variables, you should be OK.


 10:17 pm on Feb 11, 2003 (gmt 0)

2 variables after the? so the one above is to long.... right?

What about the link? I it linked from a drop down thats bad right?

What if it was a hard link to that page?



 10:54 pm on Feb 11, 2003 (gmt 0)

Actually that url has only one variable and should get spidered.


A breakdown of brand=Body%20Tone :

Variable = brand
Contents = Body%20Tone

I would ditch the %20 in your strings and use something like Body-Tone or Body+Tone but that is just a suggestion. You could also ditch the variable strings and clean up your URLs with mod_rewrite.


RewriteRule ^brand\/(.*).html /_shopByBrand.aspx?brand=$1 [L]


 11:00 pm on Feb 11, 2003 (gmt 0)

Agreed. This URL should be fine. Although I would also get rid of the URL encoded space (the %20). I replace spaces with "+" (I use the + only because that's what Google uses) and it works fine.


 11:49 pm on Feb 11, 2003 (gmt 0)


So My asp site with dynamic pages will have no problem getting spidered and listed.

So I should optimized these pages.

What about the linking anything I should know


 3:19 am on Feb 12, 2003 (gmt 0)

That's right. I get tons of pages .asp?variable=whatever included.

Sammy Jankis

 10:16 pm on Feb 18, 2003 (gmt 0)

I have been told that instead of using mod_rewrite, you can use something similar if you are running on IIS. Can anyone confirm/ elaborate on this?


 11:54 pm on Feb 18, 2003 (gmt 0)

I don't know about the mod_rewrite, but it should be pretty easy to create a custom 404 file that will do something similar. I can't remember where, but I have seen quite a few ASP based forums that do this.


I have been told that instead of using mod_rewrite, you can use something similar if you are running on IIS. Can anyone confirm/ elaborate on this?

Sammy Jankis

 12:04 am on Feb 19, 2003 (gmt 0)

Just to clarify Paladin, are you talking about a method of changing the url so that it doesn't have a query string in it? ie. (using DerekT's example)


could become


My original message might have been a bit confusing.



 12:31 am on Feb 19, 2003 (gmt 0)


would become

www.site.com/brand/Body-Tone.html which does not exist on the server, calling the 404 page.

The 404 page would then take the "brand/Body-Tone" out of the URL and treat them as if they were "?brand=Body-Tone".

I hope that clears it up.

FYI - I have done similar things in e-commerce sites. What I would usually do is have the catagory page with no querystring and the product pages with the querystring. Something like:

This made a major difference in the speed that the pages that were crawled and indexed by Google and other SE's. The catagory pages were crawled and indexed almost immediatley, while the product pages were eventually also crawled and indexed, but at least a month (if not more) later.

Something to think about.


Sammy Jankis

 1:04 am on Feb 19, 2003 (gmt 0)

I just found a company that offers a product to replace query strings with file structure urls. The product is called XQASP filter (http://xde.net/xq/tool.xqasp/qx/index.htm)

Has anyone heard of it?

(Is posting a url against the TOS?)


 3:22 am on Feb 19, 2003 (gmt 0)

I have heard it referenced a several times. The final effect is the same as changing the 404 file as I mentioned earlier, or manually coding the pages accordingly. The biggest problem that most people would have with it, along with the 404 file idea, is that they have to have their own server, as upposed to virtual server, to install it. For those that do not have access to install programs on their server, or to change the 404 file, they would have to manually code the pages to get this effect. I have done this on several sites.

From what I remember it is against the TOS to post the URL so you may want to edit it. As it is an informational link (at least partially) you may want to ask one of the forum moderators to allow it.


 8:06 pm on Feb 22, 2003 (gmt 0)

I am developing a small php content-management system. I implemented it on a small site that was previously static. At first I generated normal GET-method urls of the kind :

I soon noticed that GB would no longer index my content. So I found the "slash-separated parameters" solution on this forum, and my urls presently look like :

Needless to say, the sid is present only if the client refuses cookies, which means that it must be inserted at the entry point (so that the script can check whether the client accepts cookies or not, and not loose the session in any case).

Now, on the day I set up the modified program, I recorded *several hundred* GB hits. GB activity has now come to a more normal rate, but only the sitemap (main entry point) gets indexed...

Any idea?


 9:32 am on Feb 27, 2003 (gmt 0)

My limited understanding of the Google bot has lead me to believe that not only is it long dynamic URLs that are not indexed but also if the URL itself is long eg. more than 2 slashes in the URL as well as more than two variables in a '?' delimited URL.


 10:41 am on Feb 27, 2003 (gmt 0)

cidrolin has a separate post with that same question, see -



 3:54 pm on Feb 27, 2003 (gmt 0)

The little mantra I repeat to myself is "dynamic urls ok, avoid sessionids if at all possible." Spiders don't like sessionsids. :)


 6:14 pm on Feb 27, 2003 (gmt 0)

Thanks for the clarification. Can you or someone else tell me if this is ok. I have whittled it down to this:


No session ID. No illegal characters. :-)


 6:21 pm on Feb 27, 2003 (gmt 0)

The URL looks fine, but you also have to keep in mind that your PHP code needs to spit out good clean HTML. If it's chock full of errors, the spider won't be able to crawl it.

I assume you understand this, but thought it might bear mentioning.



 6:39 pm on Feb 27, 2003 (gmt 0)

I assume you understand this, but thought it might bear mentioning.

Thank you for checking out the URL.

Yes, very good point and one I will look into now.


 6:45 pm on Feb 27, 2003 (gmt 0)


Don't use "id=" even if it's not a session ID, there is a very good chance Google won't crawl it. I had several pages that used "id=" on one of my sites and Google wouldn't crawl them for over a year. I changed the "id=" to something else, and Google crawled them the next day.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved