Dynamic pages

Will they get spidered and listed

[jjdesigns4u]

Will Google see my dynamic pages?

Does Google ingnore the link below because it has a?
mysite.com/_shopByBrand.aspx?brand=Body%20Tone

What is the final or current say on dynamic pages and gettting them spidered and listed?

I thought Google did not follow or list dynamic pages?

Thanks

[Krapulator]

Hi There,

If you keep your querystrings to a maximum of two short variables, you should be OK.

[jjdesigns4u]

2 variables after the? so the one above is to long.... right?

What about the link? I it linked from a drop down thats bad right?

What if it was a hard link to that page?

Thanks

[DerekT]

Actually that url has only one variable and should get spidered.

mysite.com/_shopByBrand.aspx?brand=Body%20Tone

A breakdown of brand=Body%20Tone :

Variable = brand
Contents = Body%20Tone

I would ditch the %20 in your strings and use something like Body-Tone or Body+Tone but that is just a suggestion. You could also ditch the variable strings and clean up your URLs with mod_rewrite.

www.site.com/brand/Body-Tone.html

RewriteRule ^brand\/(.*).html /_shopByBrand.aspx?brand=$1 [L]

[atadams]

Agreed. This URL should be fine. Although I would also get rid of the URL encoded space (the %20). I replace spaces with "+" (I use the + only because that's what Google uses) and it works fine.

[jjdesigns4u]

OK.......

So My asp site with dynamic pages will have no problem getting spidered and listed.

So I should optimized these pages.

What about the linking anything I should know

[taxpod]

That's right. I get tons of pages .asp?variable=whatever included.

[Sammy Jankis]

I have been told that instead of using mod_rewrite, you can use something similar if you are running on IIS. Can anyone confirm/ elaborate on this?

[paladin]

I don't know about the mod_rewrite, but it should be pretty easy to create a custom 404 file that will do something similar. I can't remember where, but I have seen quite a few ASP based forums that do this.

Alon

I have been told that instead of using mod_rewrite, you can use something similar if you are running on IIS. Can anyone confirm/ elaborate on this?

[Sammy Jankis]

Just to clarify Paladin, are you talking about a method of changing the url so that it doesn't have a query string in it? ie. (using DerekT's example)

mysite.com/_shopByBrand.aspx?brand=Body%20Tone

could become

www.site.com/brand/Body-Tone.html

My original message might have been a bit confusing.

SJ

[paladin]

yes,
mysite.com/_shopByBrand.aspx?brand=Body-Tone

would become

www.site.com/brand/Body-Tone.html which does not exist on the server, calling the 404 page.

The 404 page would then take the "brand/Body-Tone" out of the URL and treat them as if they were "?brand=Body-Tone".

I hope that clears it up.

FYI - I have done similar things in e-commerce sites. What I would usually do is have the catagory page with no querystring and the product pages with the querystring. Something like:
www.mysite.com/catagory-name.asp
www.mysite.com/products.asp?id=55

This made a major difference in the speed that the pages that were crawled and indexed by Google and other SE's. The catagory pages were crawled and indexed almost immediatley, while the product pages were eventually also crawled and indexed, but at least a month (if not more) later.

Something to think about.

Alon

[Sammy Jankis]

I just found a company that offers a product to replace query strings with file structure urls. The product is called XQASP filter (http://xde.net/xq/tool.xqasp/qx/index.htm)

Has anyone heard of it?

(Is posting a url against the TOS?)

[paladin]

I have heard it referenced a several times. The final effect is the same as changing the 404 file as I mentioned earlier, or manually coding the pages accordingly. The biggest problem that most people would have with it, along with the 404 file idea, is that they have to have their own server, as upposed to virtual server, to install it. For those that do not have access to install programs on their server, or to change the 404 file, they would have to manually code the pages to get this effect. I have done this on several sites.

From what I remember it is against the TOS to post the URL so you may want to edit it. As it is an informational link (at least partially) you may want to ask one of the forum moderators to allow it.

[cidrolin]

I am developing a small php content-management system. I implemented it on a small site that was previously static. At first I generated normal GET-method urls of the kind :
[volavoile.be...]

I soon noticed that GB would no longer index my content. So I found the "slash-separated parameters" solution on this forum, and my urls presently look like :
[volavoile.be...]

Needless to say, the sid is present only if the client refuses cookies, which means that it must be inserted at the entry point (so that the script can check whether the client accepts cookies or not, and not loose the session in any case).

Now, on the day I set up the modified program, I recorded *several hundred* GB hits. GB activity has now come to a more normal rate, but only the sitemap (main entry point) gets indexed...

Any idea?

[piffdude]

My limited understanding of the Google bot has lead me to believe that not only is it long dynamic URLs that are not indexed but also if the URL itself is long eg. more than 2 slashes in the URL as well as more than two variables in a '?' delimited URL.

[aspdesigner]

cidrolin has a separate post with that same question, see -

[webmasterworld.com...]

[GoogleGuy]

The little mantra I repeat to myself is "dynamic urls ok, avoid sessionids if at all possible." Spiders don't like sessionsids. :)

[pgrote]

Thanks for the clarification. Can you or someone else tell me if this is ok. I have whittled it down to this:

[mysite.com...]

No session ID. No illegal characters. :-)

[Grumpus]

The URL looks fine, but you also have to keep in mind that your PHP code needs to spit out good clean HTML. If it's chock full of errors, the spider won't be able to crawl it.

I assume you understand this, but thought it might bear mentioning.

G.

[pgrote]

I assume you understand this, but thought it might bear mentioning.

Thank you for checking out the URL.

Yes, very good point and one I will look into now.

[atadams]

http://www.mysite.com/modules/one/myfile.php?id=1

Don't use "id=" even if it's not a session ID, there is a very good chance Google won't crawl it. I had several pages that used "id=" on one of my sites and Google wouldn't crawl them for over a year. I changed the "id=" to something else, and Google crawled them the next day.