Welcome to WebmasterWorld Guest from 54.145.144.101

Forum Moderators: open

Message Too Old, No Replies

Dynamic pages

Will they get spidered and listed

   
10:10 pm on Feb 11, 2003 (gmt 0)

10+ Year Member



Will Google see my dynamic pages?

Does Google ingnore the link below because it has a?
mysite.com/_shopByBrand.aspx?brand=Body%20Tone

What is the final or current say on dynamic pages and gettting them spidered and listed?

I thought Google did not follow or list dynamic pages?

Thanks

10:13 pm on Feb 11, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi There,

If you keep your querystrings to a maximum of two short variables, you should be OK.

10:17 pm on Feb 11, 2003 (gmt 0)

10+ Year Member



2 variables after the? so the one above is to long.... right?

What about the link? I it linked from a drop down thats bad right?

What if it was a hard link to that page?

Thanks

10:54 pm on Feb 11, 2003 (gmt 0)

10+ Year Member



Actually that url has only one variable and should get spidered.

mysite.com/_shopByBrand.aspx?brand=Body%20Tone

A breakdown of brand=Body%20Tone :

Variable = brand
Contents = Body%20Tone

I would ditch the %20 in your strings and use something like Body-Tone or Body+Tone but that is just a suggestion. You could also ditch the variable strings and clean up your URLs with mod_rewrite.

www.site.com/brand/Body-Tone.html


RewriteRule ^brand\/(.*).html /_shopByBrand.aspx?brand=$1 [L]
11:00 pm on Feb 11, 2003 (gmt 0)

10+ Year Member



Agreed. This URL should be fine. Although I would also get rid of the URL encoded space (the %20). I replace spaces with "+" (I use the + only because that's what Google uses) and it works fine.
11:49 pm on Feb 11, 2003 (gmt 0)

10+ Year Member



OK.......

So My asp site with dynamic pages will have no problem getting spidered and listed.

So I should optimized these pages.

What about the linking anything I should know

3:19 am on Feb 12, 2003 (gmt 0)

10+ Year Member



That's right. I get tons of pages .asp?variable=whatever included.
10:16 pm on Feb 18, 2003 (gmt 0)

10+ Year Member



I have been told that instead of using mod_rewrite, you can use something similar if you are running on IIS. Can anyone confirm/ elaborate on this?
11:54 pm on Feb 18, 2003 (gmt 0)

10+ Year Member



I don't know about the mod_rewrite, but it should be pretty easy to create a custom 404 file that will do something similar. I can't remember where, but I have seen quite a few ASP based forums that do this.

Alon

I have been told that instead of using mod_rewrite, you can use something similar if you are running on IIS. Can anyone confirm/ elaborate on this?
12:04 am on Feb 19, 2003 (gmt 0)

10+ Year Member



Just to clarify Paladin, are you talking about a method of changing the url so that it doesn't have a query string in it? ie. (using DerekT's example)

mysite.com/_shopByBrand.aspx?brand=Body%20Tone

could become

www.site.com/brand/Body-Tone.html

My original message might have been a bit confusing.

SJ

12:31 am on Feb 19, 2003 (gmt 0)

10+ Year Member



yes,
mysite.com/_shopByBrand.aspx?brand=Body-Tone

would become

www.site.com/brand/Body-Tone.html which does not exist on the server, calling the 404 page.

The 404 page would then take the "brand/Body-Tone" out of the URL and treat them as if they were "?brand=Body-Tone".

I hope that clears it up.

FYI - I have done similar things in e-commerce sites. What I would usually do is have the catagory page with no querystring and the product pages with the querystring. Something like:
www.mysite.com/catagory-name.asp
www.mysite.com/products.asp?id=55

This made a major difference in the speed that the pages that were crawled and indexed by Google and other SE's. The catagory pages were crawled and indexed almost immediatley, while the product pages were eventually also crawled and indexed, but at least a month (if not more) later.

Something to think about.

Alon

1:04 am on Feb 19, 2003 (gmt 0)

10+ Year Member



I just found a company that offers a product to replace query strings with file structure urls. The product is called XQASP filter (http://xde.net/xq/tool.xqasp/qx/index.htm)

Has anyone heard of it?

(Is posting a url against the TOS?)

3:22 am on Feb 19, 2003 (gmt 0)

10+ Year Member



I have heard it referenced a several times. The final effect is the same as changing the 404 file as I mentioned earlier, or manually coding the pages accordingly. The biggest problem that most people would have with it, along with the 404 file idea, is that they have to have their own server, as upposed to virtual server, to install it. For those that do not have access to install programs on their server, or to change the 404 file, they would have to manually code the pages to get this effect. I have done this on several sites.

From what I remember it is against the TOS to post the URL so you may want to edit it. As it is an informational link (at least partially) you may want to ask one of the forum moderators to allow it.

8:06 pm on Feb 22, 2003 (gmt 0)

10+ Year Member



I am developing a small php content-management system. I implemented it on a small site that was previously static. At first I generated normal GET-method urls of the kind :
[volavoile.be...]

I soon noticed that GB would no longer index my content. So I found the "slash-separated parameters" solution on this forum, and my urls presently look like :
[volavoile.be...]

Needless to say, the sid is present only if the client refuses cookies, which means that it must be inserted at the entry point (so that the script can check whether the client accepts cookies or not, and not loose the session in any case).

Now, on the day I set up the modified program, I recorded *several hundred* GB hits. GB activity has now come to a more normal rate, but only the sitemap (main entry point) gets indexed...

Any idea?

9:32 am on Feb 27, 2003 (gmt 0)

10+ Year Member



My limited understanding of the Google bot has lead me to believe that not only is it long dynamic URLs that are not indexed but also if the URL itself is long eg. more than 2 slashes in the URL as well as more than two variables in a '?' delimited URL.
10:41 am on Feb 27, 2003 (gmt 0)

10+ Year Member



cidrolin has a separate post with that same question, see -

[webmasterworld.com...]

3:54 pm on Feb 27, 2003 (gmt 0)

WebmasterWorld Senior Member googleguy is a WebmasterWorld Top Contributor of All Time 10+ Year Member



The little mantra I repeat to myself is "dynamic urls ok, avoid sessionids if at all possible." Spiders don't like sessionsids. :)
6:14 pm on Feb 27, 2003 (gmt 0)

10+ Year Member



Thanks for the clarification. Can you or someone else tell me if this is ok. I have whittled it down to this:

[mysite.com...]

No session ID. No illegal characters. :-)

6:21 pm on Feb 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The URL looks fine, but you also have to keep in mind that your PHP code needs to spit out good clean HTML. If it's chock full of errors, the spider won't be able to crawl it.

I assume you understand this, but thought it might bear mentioning.

G.

6:39 pm on Feb 27, 2003 (gmt 0)

10+ Year Member



I assume you understand this, but thought it might bear mentioning.

Thank you for checking out the URL.

Yes, very good point and one I will look into now.

6:45 pm on Feb 27, 2003 (gmt 0)

10+ Year Member



http://www.mysite.com/modules/one/myfile.php?id=1

Don't use "id=" even if it's not a session ID, there is a very good chance Google won't crawl it. I had several pages that used "id=" on one of my sites and Google wouldn't crawl them for over a year. I changed the "id=" to something else, and Google crawled them the next day.