SEO friendly URLs Part 2

[jasimon9]

---

System: The following 2 messages were cut out of thread at: http://www.webmasterworld.com/apache/4464350.htm [webmasterworld.com] by engine - 3:08 pm on Jan 21, 2013 (utc 0)

---

I am getting a strange situation with the sample code provided by lucy24 above. I first put the redirect code on a website instance on my Mac. It works as expected. However, the identical redirect code leads to an infinite loop on a Windows instance.

Here is an example of the redirect code:

RewriteCond %{THE_REQUEST} \?
RewriteCond %{QUERY_STRING} ^mode=create&usertype=pri$
RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]]
RewriteRule ^manufacturers-sales-reps-register\.php /register.php/?mode=create&usertype=pri [L,NC]

Again: working as expected on the Mac; not on Windows. Both are apache 2.2.x (x is different though: 2.2.4 on Windows and 2.2.22 on Mac).

I am wondering if there are some obvious differences on Windows.

[lucy24]

When you talk about Mac and Windows do you mean live servers (very, very small ones I assume) or MAMP vs. WAMP? I'd think it far more likely to be in Apache itself. After all, something has to have changed to make them bump the number from ..4 to ..22

:: poring over sample code ::

RewriteCond %{THE_REQUEST} \?
RewriteCond %{QUERY_STRING} ^mode=create&usertype=pri$
RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]]
RewriteRule ^manufacturers-sales-reps-register\.php /register.php/?mode=create&usertype=pri [L,NC]

Hm, let's buy time by looking at the other issues.

Rule 1 has two conditions; Rule 2 has none. But there's no need for two separate conditions. You're saying
#1 the query string has to be part of the original user request (literal ? in THE_REQUEST) and
#2 the query string has to consist of such-and-such literal text, with nothing before and nothing after. It can be easily collapsed into

RewriteCond %{THE_REQUEST} [A-Z]{3,9}\ /register\.php\?mode=create&usertype=pri\ HTTP

where [A-Z]{3,9} is probably overkill for GET or whatever single method you're dealing with. You'd only need to separate the "request" and "query string" conditions if the query string might include other pieces, or its elements might come in a different order.

If g1 were here, he would point out that as long as you're making short pretty URLs --for, ahem, a given definition of "short"-- you don't need that final .php at all.

In a redirect, always include the full protocol-plus-domain. This is where you standardize things like with-or-without www., and make sure there aren't any stray port references sneaking around.

Conversely: In a rewrite, never ever use the [NC] flag. It is fine-- though not essential-- in redirects, but here you're just begging for Duplicate Content.

RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]]

RewriteRule ^manufacturers-sales-reps-register\.php /register.php/?mode=create&usertype=pri [L,NC] 

The target of the first rule should be identical to the pattern of the second rule, and vice versa. (The "vice versa" is not true in all situations, but it is here.) Except of course the protocol-plus-domain part; that's only for redirects. Here you've got an extra / after the "php" in the rewrite version. Your server is probably going bonkers looking for that final directory.

And speaking of extras, I hope the final bracket in [NC,L,R=301]] is a typo. Otherwise I'd expect a whole new set of 500-class errors.

Now then:

the identical redirect code leads to an infinite loop on a Windows instance

When you say "infinite loop" do you mean external or internal? If it's internal, the error message comes from the server and the problem is with a rewrite. If it's external, the error message comes from the browser and the problem is with a redirect.

Oh yes and... Can I assume all of this is happening in htaccess? Otherwise you need a leading directory-slash in each pattern. I also have to assume there is more than these two rules in the htaccess, even if you're only looking at the mod_rewrite part. At a minimum, there would have to be a "RewriteEngine on" directive. Have you eliminated all other redirects and rewrites as possible troublemakers? Does anything in either setup use mod_alias along with mod_rewrite?

[jasimon9]

Thanks for your swift and detailed response. Almost more than one can hope for these days!

I am talking about servers. In fact I omitted that the same set of rewrite rules is also working perfectly on a development server which is on FreeBSD, Apache 2.2.21. Our production server is also FreeBSD, Apache 2.2.19. Normally we try to keep everything at the same level, but this time they have gotten a bit out of sync.

The Mac environment is "pre-development" where I was just doing some testing. Unfortunately, the Windows environment is where I debug, as I have not succeeded in migrating away from that yet.

In any case, I came up with the 4 lines from your "canonical" reposting at [webmasterworld.com...] dated 10:27 am on June 13, 2012. In that posting you presented Part 1, Part 2 (and Part 0), as follows:

Part 1
RewriteCond %{THE_REQUEST} \?
RewriteCond %{QUERY_STRING} queryname=([a-z]+)
RewriteRule longcomplicatedURL http://www.example.com/blahblah/%1? [R=301,L]

Part 2
RewriteRule blahblah/([a-z]+)$ longcomplicatedURL?queryname=$1 [L]

So I concluded that both parts are needed, giving the 4 lines that I have, adapted to my situation. I realize that you were trying to illustrate the principle as you noted that "in real life it will usually be a little more complicated, but that's the basic process."

My situation is not typical in that I do not need to take part of the query string in the regex and use it in the rewrite. However, I was hoping that the same method would be applicable.

As an aside, because I do not require the regex replacement, another approach would be to just rename the actual pages, and use a redirect from the old name to the new name. This could be simpler, but goes into other SEO areas I am not fully comfortable with. Plus there are variations in which other GET vars are used in addition to what is handled by the redirect. Yet if I became comfortable with this approach, it could be viable.

Regarding the duplicate trailing brackets, I am embarrassed to say that in fact they were place in the Mac and FreeBSD instances, but not in the Windows instance. Which means that while they are wrong, they are not causing the looping. After removing them, the Mac and FreeBSD instances continue to function as expected. Windows instance, being unchanged, still loops.

Regarding transforming "long complicated query strings" into "short pretty ones": I am actually transforming some fixed query strings into a fairly long but keyword rich filenames. The point is that the original names to not tell the search engines anything about the content.

Point taken about the added NC.

The extra / was put in during initial debugging of the Windows version.

The "infinite loop" is referring to internal. In fact, the looping is visible in the rewrite log.

None of this is using htaccess; everything is in httpd-vhosts.conf. I tried both with and without the leading directory. On our production servers there are many other rewrite rules which have been in place for a long time and work fine. mod_alias is not in use. Those other rewrite rules were put in place by an expert sysadmin with long experience. They do not have leading directory slashes in any of the patterns, nor do they have the protocol plus domain. Here is an example of one of those rules, which has been in effect since 2006:

RewriteRule ^main.asp /index.php [L,NC,R]

However, on my test servers on the Mac and Windows, there are no other rewrite rules. Only the ones I am trying to develop now. So I don't think there is any interference for other rules.

You gave a lot of information, some of which I have assimilated; other parts I am not sure I have the understanding.

Bottom line of my current understanding: the following original sample should be changed from

RewriteCond %{THE_REQUEST} \?
RewriteCond %{QUERY_STRING} ^mode=create&usertype=pri$
RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]]
RewriteRule ^manufacturers-sales-reps-register\.php /register.php/?mode=create&usertype=pri [L,NC]

to the following:

RewriteCond %{THE_REQUEST} [A-Z]{3,9}\ /register\.php\?mode=create&usertype=pri\ HTTP
RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]
RewriteRule ^/manufacturers-sales-reps-register\.php http://www.example.com/register.php?mode=create&usertype=pri [L]

Specifically, the changes here the following:

1. Combine the two conditions into one.
2. Correct the typo of the two ]] in the first rewrite rule
3. Add a leading slash to the pattern of the second rewrite rule (because using a vhost not htaccess)
4. Remove the trailing slash from the target in the second rule
5. Add the protocol and domain to the second rewrite rule (with the actual domain in place of "example")
6. Remove the NC from the second rewrite rule

Thank you again!

[jasimon9]

After making the changes described in my previous post, the page still gave an error. I jumped to the conclusion that it was still looping. However, this is apparently not always the case. Instead there is a problem with the rewriting, but no looping, as you can see from the following log entries:

192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73ca98/initial] (2) [perdir F:/Websites/RepHunter/current/] rewrite 'register.php' -> '/manufacturers-sales-reps-register.php?'
192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73ca98/initial] (2) [perdir F:/Websites/RepHunter/current/] explicitly forcing redirect with http://192.168.1.106:8080/manufacturers-sales-reps-register.php
192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73ca98/initial] (1) [perdir F:/Websites/RepHunter/current/] escaping http://192.168.1.106:8080/manufacturers-sales-reps-register.php for redirect
192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73ca98/initial] (1) [perdir F:/Websites/RepHunter/current/] redirect to http://192.168.1.106:8080/manufacturers-sales-reps-register.php [REDIRECT/301]
192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73ca98/initial] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/manufacturers-sales-reps-register.php
192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73f400/initial/redir#1] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/error404.php

Again, here are the rules now in effect copied from httpd-vhosts.conf:

RewriteCond %{THE_REQUEST} [A-Z]{3,9}\ /register\.php\?mode=create&usertype=pri\ HTTP
 RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]
 RewriteRule ^/manufacturers-sales-reps-register\.php http://www.example.com/register.php?mode=create&usertype=pri [L]

As you can see from the log entries, the final rewrite is not resolving to the page, and the error page is shown.

As a test, I removed the leading slash on the second rewrite rule. After doing do, the loop is produced. Putting the slash back in gets rid of the loop, but still results in the 404 error. So it seems that the leading slash in the pattern of the second rewrite rule is critical on the Windows instance but not on FreeBSD or Mac.

However, I tested with the revised rules on the Mac and FreeBSD instances, and now all of them produce the 404 error on trying to find manufacturers-sales-reps.php. So at least we are consistent with the same failure on all three platforms. There must be something simple that I am missing.

[g1smd]

RewriteCond %{THE_REQUEST} [A-Z]{3,9}\ /register\.php\?mode=create&usertype=pri\ HTTP
RewriteRule ^register\.php$ /manufacturers-sales-reps-register.php? [NC,L,R=301]

RewriteRule ^/manufacturers-sales-reps-register\.php http://www.example.com/register.php?mode=create&usertype=pri [L]

There are several errors in the above code.

The target of the external redirect should contain protocol and hostname. Try this:

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /register\.php\?mode=create&usertype=pri\ HTTP/
RewriteRule ^register\.php$ http://www.example.com/manufacturers-sales-reps-register.php? [R=301,L]

The target of the internal rewrite should not include the protocol or hostname. Including those turns the rule into a 302 redirect. The pattern for the rewrite should not begin with a leading slash. Try this:

RewriteRule ^manufacturers-sales-reps-register\.php /register.php?mode=create&usertype=pri [L]

Consider NOT including the .php bit in the friendly URL.

Add a blank line after every rule to make the code more readable.

[jasimon9]

Thanks for your input g1smd. I have taken your suggestion. On Windows, the absence of the leading slash causes a loop. When I put the slash back in, no loop occurs, but I get the 404 error.

lucy24 said that for a vhost (not htaccess) the leading slash is needed in each pattern. But when I put the leading slash in the first rewrite rule, the page is not redirected on the Mac instance. All my instances by the way are in a vhost <Directory>. Don't use htacess at all.

Whether the leading slash is present seems to make a huge difference, and causes the loop on Windows if missing.

Bottom line: it still does not work on Windows. A workaround would be to just dump the Windows environment. However, I have been trying to that for several years, and just cannot get a development environment I like native to the Mac, although PHPStorm seems to come the closest.

Here are the rules now in effect on Windows (same elsewhere, except when not on Windows, I have removed leading slashes from patterns):

 RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /register\.php\?mode=create&usertype=pri\ HTTP/

 RewriteRule ^/register\.php$ http://192.168.1.106:8080/manufacturers-sales-reps-register? [R=301,L]
 
 RewriteRule ^/manufacturers-sales-reps-register /register.php?mode=create&usertype=pri [L]

[g1smd]

Ah yes, I missed the fact this code is in

vhosts

not in

htaccess

. So,

RewriteRule ^/

etc, for use in

vhosts

Your code looks OK. Make sure you clear the browser cache before retesting. Is the IP address correct?

[jasimon9]

As another test I changed the second rewrite rule to have $ at the end of the pattern. With this, there seems to be no effective rewriting or redirection at all. There is no looping, but entering the "old complex" URL gives the page and shows the "old complex" URL in the address bar. Entering the "short" URL then gives a 404 error. It is like adding the $ has nullified the effect of all the rules.

Checking [httpd.apache.org...] I find the following:

The removed prefix always ends with a slash, meaning the matching occurs against a string which never has a leading slash. Therefore, a Pattern with ^/ never matches in per-directory context.

Therefore I am thinking they have to go. So the problem is just that the Windows instance has the looping problem.

[jasimon9]

IP address is correct.

Seems that on *nix "it just works"; on Windows "it just fails".

Simple rewrites do work on Windows, so I know at least something is working there. Any known gotchas that makes Windows different?

[jasimon9]

I have discovered part of my problem. One of the pages I am trying to redirect is via https. There are two parts to the problem:

1. The redirect seems to work if I change the protocol in the rule, but the rewrite still fails, leading to a 404 error.

2. Once the browser is put into https mode, all the other rewrites fail. Before having rewrites, it was not a problem to be in https mode .

[g1smd]

You probably need separate rules for http and https - especially for the redirects.

[jasimon9]

I am thinking that the way the page is put into https is the problem. The php code tests if the page is in https, and if not, redirects it using REQUEST_URI. So that is probably causing the loop or other problems. I think I have to resolve this aspect before working on the rules further.

But yes, probably separate handling would be simplest.

[g1smd]

For redirects you should usually test THE_REQUEST rather than REQUEST_URI.

REQUEST_URI is updated by internal rewrites.

[jasimon9]

Actually I should have made more clear that within PHP I am accessing $_SERVER['REQUEST_URI']. There is no THE_REQUEST within the PHP server vars.

I am in a bit of a pickle (aka catch-22) as my dev environment is in Windows, and the whole redirect thing fails in my windows environment. So I am using this as a goad to abandon windows, as I cannot debug now.

While I am at is, here is a sample of what is written to the rewrite log on windows, when it loops (which does not occur on *nix):

192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (2) [perdir F:/Websites/RepHunter/current/] rewrite 'search-by-keyword.php' -> 'http://192.168.1.106:8080/independent-sales-reps-finder.php?'
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (2) [perdir F:/Websites/RepHunter/current/] explicitly forcing redirect with http://192.168.1.106:8080/independent-sales-reps-finder.php
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (1) [perdir F:/Websites/RepHunter/current/] escaping http://192.168.1.106:8080/independent-sales-reps-finder.php for redirect
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (1) [perdir F:/Websites/RepHunter/current/] redirect to http://192.168.1.106:8080/independent-sales-reps-finder.php [REDIRECT/301]
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (2) [perdir F:/Websites/RepHunter/current/] rewrite 'independent-sales-reps-finder.php' -> '/search-by-keyword.php?usertype=pri'
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (1) [perdir F:/Websites/RepHunter/current/] internal redirect with /search-by-keyword.php [INTERNAL REDIRECT]
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#7856d0/initial/redir#1] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/search-by-keyword.php
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#78b718/initial] (2) [perdir F:/Websites/RepHunter/current/] rewrite 'independent-sales-reps-finder.php' -> '/search-by-keyword.php?usertype=pri'
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#78b718/initial] (1) [perdir F:/Websites/RepHunter/current/] internal redirect with /search-by-keyword.php [INTERNAL REDIRECT]
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767618/initial/redir#1] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/search-by-keyword.php
192.168.1.205 - - [22/Jan/2013:18:03:29 --0600] [192.168.1.106/sid#697c70][rid#767650/initial] (2) [perdir F:/Websites/RepHunter/current/] rewrite 'independent-sales-reps-finder.php' -> '/search-by-keyword.php?usertype=pri'

and on and on

And here is a sample of what happens when the rewrite fails:

192.168.1.205 - - [22/Jan/2013:18:04:36 --0600] [192.168.1.106/sid#697c70][rid#789700/initial] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/independent-sales-reps-finder.php
192.168.1.205 - - [22/Jan/2013:18:04:36 --0600] [192.168.1.106/sid#697c70][rid#78c050/initial/redir#1] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/error404.php

[g1smd]

Run the Live HTTP Headers extension for Firefox and see what that shows.

I assume the internal rewrite might also need the [QSA] flag.

[lucy24]

Overlapping last two (at least!) posts:

I changed the second rewrite rule to have $ at the end of the pattern. With this, there seems to be no effective rewriting or redirection at all.

In the pattern, $ is an anchor. It means that the text you're looking at must be the very last thing in the URL-- not counting query, if any.

Did you ever decide whether your URLs should or should not end in a slash? Since it isn't a "real" directory, you have to enforce a format; mod_dir won't do it for you. Or, uhm, am I thinking of another thread? There were two fairly similar questions side by side...

If your URL ends in a slash and the rewrite/redirect pattern doesn't, then the rule will only work if you leave off the anchor.

192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73ca98/initial] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/manufacturers-sales-reps-register.php

192.168.1.205 - - [21/Jan/2013:12:38:44 --0600] [192.168.1.106/sid#698560][rid#73f400/initial/redir#1] (1) [perdir F:/Websites/RepHunter/current/] pass through F:/Websites/RepHunter/current/error404.php

Ouch. This is a logging level I don't read fluently, but this is the core of the problem, right? At this point your server is looking for a physical file called

current/manufacturers-sales-reps-register.php

which of course doesn't exist, because the newly redirected request is now supposed to get rewritten.

With opening anchors you can always cheat by leaving off both the ^ and the leading / and then you don't have to remember whether the / is supposed to be there or not. Then, once you've got everything else working, add the slash-plus-anchor combination so the rule will run faster in real life. Opening anchor means mod_rewrite may only have to look at one or two characters to see if the rule fails. Otherwise it has to read the whole thing.

[jasimon9]

Thanks to lucy24 and g1smd for your helpful comments regarding the anchors, as well as all your previous help.

However, I think I now have the core of the problem, but not the solution. It turns out that the page that is having the problem, due to PCI Compliance requirements, needs to be served over SSL. That is, using https. However, we don't support SSL on all of our quality domains (namely, various workstations and development servers). The way that this has been implemented therefore is that at the PHP level, the code checks whether the page should be using SSL, and if not redirects it. The PHP code is as follows:

// force ssl to protect credentials
if (ENABLE_FORCE_SSL && !$_SERVER['HTTPS'])
{
 if ($external_link)
 {
 $_SESSION['site_user_type'] = '';// enable redirect to keep external link
 }
 GoToPage(substr($RH_URL_SSL,0,-1) . $_SERVER['REQUEST_URI']);
}

This in part explains why on certain instances no problem occurs.

The ENABLE_FORCE_SSL flag is enabled on a per domain basis. Debugging shows that the contents of the server var $_SERVER['REQUEST_URI'] is the redirected page name "/manufacturers-sales-reps-register.php". The GoToPage function does some processing and ends with sending the "Location" header to do the redirect. The argument to the Location would in this case be [example.com...]

So I am thinking at this point that is where the failure occurs. Unfortunately I cannot trace the failure mode at the present time because the development environments don't support SSL. So I am a bit stuck.

[jasimon9]

Took me a few days to get back to this issue.

The comment from g1smd about having separate rules for https in fact appears to solve the problem. I updated the appropriate section in httpd-ssl.conf.

Kudos to g1smd and lucy24 for being most helpful!

[g1smd]

Glad you fixed it.

It's often guesswork as to what the problem is.

The more clues in the question, the more guesses we can make. :)