URLs for search engine crawls

rewriting urls to make them search engine friendly

[Yosha]

I am currently using mod_rewrite to make my urls search engine friendly for my ecommerce site. I am trying to eliminate all .cgi, =, & out of the url. However some of them are getting out of hand with blank areas where variables are not filled in.

http*//example.com/prod/123//n/abc////Y//def

These variables would be filled in during a normal shopping experience so I have to keep them around. Now my question, is this going to cause a problem with having my site crawled? The link above would be similar to a link to one of my products. I haven't found any information about the affects of having two slashes next to each other. Would it help if I converted the slashes to commas? Please help!

[edited by: pageoneresults at 7:03 pm (utc) on Nov. 12, 2003]
[edit reason] Delinked URI [/edit]

[pageoneresults]

Hmmm, interesting situation you are in. I can only take a wild guess at this one. The spider will look at each one of those / as a sub-directory. Having two of them back to back is something I've not seen in my tenure.

I would concentrate on eliminating those / as they are probably going to cause issues. This URI should be shortened as much as possible...

http*//example.com/prod/123//n/abc////Y//def

http*//www.example.com/prod/123/

I might even go one step further and shorten it to...

http*//www.example.com/123/

... but only if you can. When rewriting URIs, the goal is to eliminate as many of the variables as possible so that you end up with a short user friendly URI. This...

http*//example.com/prod/123//n/abc////Y//def

... is not going to work from my perspective. You've got at least 10 levels there and I doubt very seriously that any well behaved spider is going to go that deep any time soon. Googlebot may get to the content at some point but my experience has shown that the shorter the URI, the quicker the page gets indexed and the more user friendly it is.

[illudium]

I don't completley understand the question, but make the slashes, dashes instead.

You could also try fixing your rewrite to expand certain single slashes to stacked internal slashes. I would be very wary of system resources when doing mult. rewrite levels.

and the above about it being too deep for google. hooey. Just get your urls in a non-dynamic form and google will own you.

[Yosha]

I wish that I could eliminate all the extra space in my url but some of the variables are necessary in order to complete a transaction. (delicate balance b/w being crawled and having a functioning shopping cart). I have however managed to cut some of them out.

Would there be a complication if I substituted some of the slashes for hyphens?

www.example.com/prod/123/usr-rrc-act

or it could be

www.example.com/prod/123/--

I still seeing if it's feasible to eliminate the prod, since it is being used for my pattern matching.

[edited by: agerhart at 2:53 pm (utc) on Nov. 18, 2003]
[edit reason] delinked [/edit]

[barn_de]

Hi Yosha,

i agree with pageoneresults. try to remove all the / from the url.

you can still have the pattern matching for the products if you just put something in the url which is unique on a product page.

eg:

www.yoursite.com/prod-123
so the rewrite rule would look like this:

^/prod-(.*)$ points to /product.php?ProdId=$1

and if you need the more variables just use something like this:
www.yoursite.com/prod-123-var1-cc

and the rewrite:

^/prod-(.*)-var1-(.*)$ points to /product.php?ProdId=$1&Var1=$2

i'm not sure, if the rewrite rules i posted work 100%, but i think this gives you an idea, how to solve this problem.

barn

[edited by: agerhart at 2:54 pm (utc) on Nov. 18, 2003]
[edit reason] delinked [/edit]

[Yosha]

That's what I'm going to try to do but there is one thing that I'm not sure of...

When someone is in my shopping cart they are automatically give a customer number. If a search engine crawls my site, will it too get a customer number? If so all is fine and dandy, if not then I may have empty variables.

www.example.com/prod-123/-

ie. www.example.com/prod-123/[customer#]-[email addr]

These two variables are essential to the "customer experience.