SES rewrite, tricky url variables, reverse-writing old urls

[zsa zsa gabor]

This is a bit similar to [webmasterworld.com...] except more complicated and I'm less clever.

I made a new rewrite rule to change dynamic urls to short static urls of the form "item123" that works but maybe isn't the best:

RewriteRule ^item([0-9]*[^/\.])$ product.php?product_id=$1 [qsa,L]

I copied the [^/\.] part somewhere; I assume it prevents a directory or file name hit.

Extra url variables I wanted to get tacked on using [qsa]. Url variables, for example, might be ?currency=EUR&displaymode=2 and such. Or for a similar category rewrite it can be &startrow=51, etc.

The big problem is I want to 301 redirect the old dynamic urls coming in to the new style. I think it goes before the new rewrite with a [NC,R=301,L] and maybe qsa, but:

1) how to parse out the product_id to use as the page name?

2) then how to tack on all the possible url variables EXCEPT the product_id one? If using [qsa] it will add product_id to the url and be wrong I assume, like item123?product_id=123&currency=EUR.

For example redirect old product.php?product_id=123&currency=EUR&displaymode=2 to item123?currency=EUR&displaymode=2

Suggestions on best handling the 1st issue and doing the 2nd?

Thanks.

[edited by: zsa_zsa_gabor at 6:07 pm (utc) on Aug. 1, 2009]

[jdMorgan]

> For example redirect old product.php?product_id=123&currency=EUR&displaymode=2 to item123?currency=EUR&displaymode=2

Test %{THE_REQUEST} to be sure that this request is coming from a client and not as a result of your internal rewrite, and also to capture any query-string name/value pairs that precede or follow the product_id:

# Check client request: Accept optional parm(s) before product_id name/value, but none after
RewriteCond %{THE_REQUEST} ^[A-Z]+\ /product\.php\?(([^&]*(&[^&]*)*)&)?product_id=([0-9]+)\ HTTP/ [OR]
# Check client request: Require parm(s) before and accept optional parm(s) after product_id name/value
RewriteCond %{THE_REQUEST} ^[A-Z]+\ /product\.php\?(([^&]*(&[^&]*)*)&)product_id=([0-9]+)((&[^&\ ]*)*)\ HTTP/
RewriteRule ^product\.php$ http://www.example.com/item%4?%2%5 [R=301,L]

Note that THE_REQUEST is the exact client request line, as it appears in your raw server access log file. For example:

GET /product.php?foo=bar&product_id=123&currency=EUR&displaymode=2 HTTP/1.1

Jim

[edited by: jdMorgan at 9:50 pm (utc) on Aug. 1, 2009]

[g1smd]

You assume correct about not matching folders or files.

Make sure the redirect contains the domain name in the target URL, and that all redirects are listed before all rewrites.

[zsa zsa gabor]

>>Accept optional parm(s) before product_id

I don't think that is a concern. If not, can just go with the 2nd line, correct?

I will let you know the result after I try. Thanks for the suggestions.

One thing I don't understand about the rewrite (#1 above) is why this didn't work, seems it should catch item123:

RewriteRule ^item([0-9]*)$ product.php?product_id=$1 [qsa,L]

I think I tried that separately, but also I'm putting this in httpd.conf and I've read you need a slash first, but this didn't work either:

RewriteRule ^/item([0-9]*[^/\.])$ product.php?product_id=$1 [qsa,L]

[zsa zsa gabor]

After looking at it more closely, jd, I think I need to change yours. The product_id is always first, so the 2nd line will never work, correct?

Also, isn't your code always going to stick a ? on the redirect url even if %2%5 are empty? How to make that ? dependent on there being variables in %2%5?

Can I make the before variables merely possible like the first line (just in case)? Why then can't I have just one line like this - though I'm not sure how to count the () variables, so I'm just counting the outer ones for this example and know that is wrong:

# Check client request: OPTIONAL parm(s) before and accept optional parm(s) after product_id name/value
RewriteCond %{THE_REQUEST} ^[A-Z]+\ /product\.php\?(([^&]*(&[^&]*)*)&)?product_id=([0-9]+)((&[^&\ ]*)*)\ HTTP/
RewriteRule ^product\.php$ http://www.example.com/item%2?%1%3 [R=301,L]

And I assume I'm wrong but were you missing a ? after the last () to make those optional - otherwise there must be an & present -, so must be:

RewriteCond %{THE_REQUEST} ^[A-Z]+\ /product\.php\?(([^&]*(&[^&]*)*)&)?product_id=([0-9]+)((&[^&\ ]*)*)?\ HTTP/
RewriteRule ^product\.php$ http://www.example.com/item%2?%1%3 [R=301,L]

Can one line not handle everything? Don't quite understand why you have 2 lines if the end variables are optional.

If you feel like teaching, I also don't understand why you have ((&[^&\ ]*)*) on the end. I would think (&[^\ ]*)? would be sufficient to capture all the end up to the space if present whereas yours would stop at the end of the first variable pair - "^&" - and has more brackets.

Also, I assume changing to [NC,R=301,L] is fine?

Thanks again for your help. I've never used mod_rewrite before and regex only a bit.

[zsa zsa gabor]

WORKING!

I should add I tried your code as-is, and fiddling with the 2nd line to make the before variables optional or to remove that part altogether, but couldn't get it to work. Nor did my noodlings noted above.

This is what I currently have working, I realized I had the NC on the wrong line: get the ID numbers in %1, maybe an "&" which we don't want, then get anything or nothing after that up to a space in %2:

#RewriteCond %{THE_REQUEST} ^[A-Z]+\ /product\.php\?product_id=([0-9]+)&?([^\ ]*)\ HTTP/ [NC]
#RewriteRule ^product\.php$ http://example.com/item%1?%2 [R=301,L]

Any comments about unseen problems here?

I still don't understand why the item%1?%2 doesn't output an unwanted "?" on the end when no %2 variable present.

[jdMorgan]

When working with regular expressions and mod_rewrite, be aware that characters have vastly-different meanings depending on whether (and how and where) they are used in regex patterns or in other parts of a rule.

In a regex pattern, a "?" is a quantifier meaning "match zero or one of the preceding character or parenthesized sub-pattern" unless that question mark appears in an alternate-character group enclosed by "[]" or is preceded by a backslash "\", in which case it is treated as a literal "?" character to be matched.

A question mark appearing at the send of a substitution URL or filepath in a RewriteRule is a token that means "replace the current query string" unless the [QSA] flag is used, in which case it means "append this to the current query string." However if no characters follow that "?", then the current query string is simply replaced with blank, and the question mark does not appear in the output URL or filepath.

Earlier questions:

> I think I tried that separately, but also I'm putting this in httpd.conf and I've read you need a slash first, but this didn't work either:

If the code is located in a conf file outside of a <Directory> section, it needs a leading slash. Within a <Directory> section, or within a .htaccess file, it does not need a slash, because the req_rec is 'localized' to the current directory.

> After looking at it more closely, jd, I think I need to change yours. The product_id is always first, so the 2nd line will never work, correct?

You can do that if you're absolutely sure that the product_id will always be first. I couldn't be sure, so I provided a "robust" solution. Frankly, when working in a "forum" environment, it's best to test the code exactly as posted, to avoid the confusion of a "constantly moving target" and save time. Get the code working as-posted first, then you can tweak it as much as you like without having to post and ask about every change in the forum -- and you'll always have the working example to revert to.

> I'm not sure how to count the () variables
Count left parentheses to resolve back-reference numbers.

> And I assume I'm wrong but were you missing a ? after the last () to make those optional - otherwise there must be an & present -, so must be:

Nope. They're already optional, because the quantifiers at both "levels" are "*" -- meaning "match any number, including zero."

> Can one line not handle everything? Don't quite understand why you have 2 lines if the end variables are optional.

No because you could end up with an orphaned or doubled "&" if you do that.

> If you feel like teaching, I also don't understand why you have ((&[^&\ ]*)*) on the end.

So that the back-references always refer to the "correct" part of *both* RewriteConds, no matter which one is matched. That is, the number and placement of parentheses in both RewriteConds must be the same, or the rule won't use the correct 'pieces' of the input query string in the rewritten URL/filepath.

Jim

[zsa zsa gabor]

>>Within a <Directory> section... it does not need a slash

My case.

>>the number and placement of parentheses in both RewriteConds must be the same

Figured that must be it. Do we just count from left to right to know which number to refer to, or left-right but also inside-out?

>>it's best to test the code exactly as posted
As mentioned, I did try your code but didn't work. Though could have been the missing [NC].

If you can see any problem with my one line solution, please let me know. Seems to be working fine in all cases, with 0, 1 or 2 variables added on the end.

Thanks for explaining a few things. Very kind of you.