mod rewrite for domain and subdirectory changes

Redirecting requests from superseded bookmarks

[Carob]

To accommodate changes in terminology, I have changed the domain name and various subdirectories in my web site. The old domain name 'oldexample.com' is a parked domain pointing to the same IP address as 'newexample.com', but not having a specified redirection in cPanel (at present).

The .htaccess file I have in the public_html directory is as follows (edited down for brevity):
--------------------
Options +FollowSymlinks
Options -Indexes

<FilesMatch \.[Hh][Tt][AaPpGg].+$">
Order allow,deny
Deny from all
</FilesMatch>

Addhandler application/x-httpd-php5 .html .php

RewriteEngine on
RewriteBase /

# Rule 1
# Block useless bots
RewriteCond %{HTTP_USER_AGENT} ^(.*)Baiduspider(.*) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(.*)YandexBot(.*) [NC]
RewriteRule ^(.*)$ - [F,R=403,L]

# Rule 2
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^newexample\.com$ [NC]
RewriteRule ^(.*)$ [newexample.com...] [R=301,QSA]

# Rule 3
# Redirect to HTTP for non-secure pages
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{SERVER_PORT} 443
RewriteCond %{REQUEST_URI} ^/index.html$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/afa/index.html$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/pf1/index.html$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/pf2/index.html$ [NC]
RewriteRule ^(.*)$ [%{HTTP_HOST}...] [R=301]

# Rule 4
# Redirect those seeking 'als' to 'afa' and 'sf1' to 'pf1' and 'sf2' to 'pf2' and 'sfa' to 'pf2'
RewriteCond %{REQUEST_URI} /(als|sf[12a]) [NC]
RewriteRule /als(.*)$ /afa$1 [NC,QSA,L]
RewriteRule /sf([1-2])(.*)$ /pf$1$2 [NC,QSA,L]
RewriteRule /sfa(.*)$ /pf2$1 [NC,QSA,L]

--------------------

Rule 2 is intended to direct requests from:
www.oldexample.com
oldexample.com
www.newexample.com
to newexample.com, and appears to work.

Rule 3 works, but I include it here only because it may have an influence on the working of Rule 4.

Rule 4 is the problem area. I wish to rewrite requests for /als to /afa and for /als/index.html to /afa/index.html, etc. If the domain of the request is any one of www.oldexample.com, oldexample.com, or www.newexample.com, and the request doesn't specify index.html, then the redirect works. If the domain is newexample.com, the result is a 404 Not Found error, and if index.html is specifically required, then domain/index.html/index.html results.

I have persisted in trying to solve these issues for quite some time, mostly referring to WebmasterWorld as the authoritative source, but I have not succeeded in fixing the .htaccess file. The 'experts' at my hosting service intervene when I have asked, but their changes haven't been any help and they may be out of their depth. A complicating issue is that redirects specified in my .htaccess file seem to be inherited into cPanel after a time, and I don't know at what level the cPanel redirects are called, or whether having redirects in two places is a problem.

I would very much appreciate some help.

[phranque]

which is the directive that made this happen?

GET /private/admin/test/links-index.html HTTP/1.1
...
HTTP/1.1 301 Moved Permanently
...
Location: http://example.com/401.shtml

[lucy24]

Yes, that's what I meant before but wasn't clear enough. I thought you were for some reason asking for "401.shtml" by name, which is odd but OK. The full version from Firebug means that someone has been explicitly redirected to it. And "401.shtml" doesn't even sound like the name of a validation page; it sounds like an error document. Aside from anything else, why is it .shtml? That would never be a default extension.

[Carob]

'phranque': The request was for "http://oldexample.com/private/admin/test/links-index.html".

The applicable rule:

# Rewrite to HTTPS for secure 'other'.html pages in /private/admin/test/
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^((?:characters|course-data-read|links-index)\.html)$ [example.com...] [R=301,L]

This produced the '301' status code correctly. From the 'Live HTTP Headers' result I quoted earlier (#4672968 at 8:30 am on May 21, 2014), it appeared there was a call for authentication. But there was not an authentication pop-up window delivered to the browser; I don't know why.

Certainly the file "/private/admin/test/links-index.html" exists. I do not know why it then sought "Location: http://example.com/401.shtml"; I was hoping you or Lucy could tell me. There is no such file and I did not request it.

Lucy: Immediately after I saw your post (#4672935 5:54 am on May 21, 2014), I put in place "/art/404-not-found.html" with the .htaccess entry "ErrorDocument 404 http://example.com/art/404-not-found.html". The same request now finds that page successfully.

I do not understand why the rule I quoted above does not redirect the request correctly: it gets neither the "https://..." part nor the file that exists in the subdirectory.

As to the order the the mod_rewrite rules, I share Marcia's confusion (http://www.webmasterworld.com/apache/3524863.htm). I was heartened to hear from Lucy in her first reply, "On the plus side, the quoted rules seem to be in the right order", and I have since reordered specific rules as she has suggested, including moving the canonicalisation to the last place in the .htaccess file, where it seems almost, but not totally, unnecessary, given the rules that now occur before it.

At the risk of making another excessively long post, but to avoid confusion, the .htaccess file now looks like this:

Options +FollowSymlinks
Options -Indexes

<Files "robots.txt">
Order Allow,Deny
Allow from all
</Files>

Addhandler application/x-httpd-php5 .html .php

ErrorDocument 404 http://example.com/art/404-not-found.html

RewriteEngine on
RewriteBase /

# (1) Block useless bots
RewriteCond %{HTTP_USER_AGENT} (Baiduspider|YandexBot) [NC]
RewriteRule . - [F]

# (2) Rewrite those seeking 'als' to 'afa', 'bookings1' to 'bookings2', 'sf1' to 'pf1', 'sf2' to 'pf2', and 'sfa' to 'pf2'
RewriteRule ^als(.*)$ http://example.com/afa$1 [R=301,L]
RewriteRule ^bookings1(.*)$ [example.com...] [R=301,L]
RewriteRule ^sf([12].*)$ http://example.com/pf$1 [R=301,L]
RewriteRule ^sfa(.*)$ http://example.com/pf2$1 [R=301,L]

# (3) Rewrite to HTTP for non-secure 'other'.html pages at the root
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{SERVER_PORT} 443
RewriteRule ^((?:tour|example[-a-z]+)\.html)$ http://example.com/$1 [R=301,L]

# (4) Rewrite to HTTP for non-secure 'other'.html pages in /art/
# RewriteCond %{HTTPS} =on [OR]
# RewriteCond %{SERVER_PORT} 443
# RewriteRule ^((?:banner|buttons-row[123]|contacts)\.html)$ http://example.com/art/$1 [R=301,L]

# (5) Rewrite to HTTPS for secure 'other'.html pages in /bookings2/
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^((?:booking-entry|booking-save)\.html)$ [example.com...] [R=301,L]

# (6) Rewrite to HTTPS for secure identity.html page in /private/history/
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^identity\.html$ [example.com...] [R=301,L]

# (7) Rewrite to HTTPS for secure bookmarks.html page in /private/reference/
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^bookmarks\.html$ [example.com...] [R=301,L]

# (8) Rewrite to HTTPS for secure buttons-row4.html page in /private/admin/
# RewriteCond %{HTTPS} =off [OR]
# RewriteCond %{SERVER_PORT} !443
# RewriteRule ^buttons-row4\.html$ [example.com...] [R=301,L]

# (9) Rewrite to HTTPS for secure 'other'.html pages in /private/admin/test/
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^((?:characters|course-data-read|links-index)\.html)$ [example.com...] [R=301,L]

# (10) Rewrite to HTTP for non-secure 'other'.html pages
# RewriteCond %{HTTPS} =on [OR]
# RewriteCond %{SERVER_PORT} 443
# RewriteRule ^((?:art)/.*)?$ http://example.com/$1 [R=301,L]

# (11) Rewrite to HTTPS for secure 'other'.html pages
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^((?:bookings2|private/history|private/reference|private/admin|private/admin/test).*) [example.com...] [R=301,L]

# (12) Rewrite to HTTP for non-secure index.html pages
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{SERVER_PORT} 443
RewriteRule ^((afa|als|art|bookings|classes|els|faqs|fees|forms|gallery|pf1|pf2|rfa|unused)/)?index\.html http://example.com/$1 [R=301,L,NS]

# (13) Rewrite to HTTPS for secure index.html pages
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443
RewriteRule ^((admin|bookings2|private|private/history|private/reference|private/admin|private/admin/counter|private/admin/file-log|private/admin/files|private/admin/page-log|private/admin/test|private/trainers)/)index\.html [example.com...] [R=301,L,NS]

# (14) Host Name Canonicalisation (use non-www form of new domain)
RewriteCond %{HTTP_HOST} !^(example\.com)?$ [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]

I realise there are parts here that could be expressed more succinctly with better regular expressions, but I thought it best to get the functionality correct before fine-tuning. Excepting for the addition of the '404' file, the file and directory structure is still as I listed earlier.

Thanks for your help.

[lucy24]

almost, but not totally, unnecessary

:) The rule is for insurance. In practice the chances are pretty remote that someone will ask for an exactly right pagename, getting only the www. wrong. It could happen if someone else links to you, and in the course of editing they add or remove a "www." Search engines will also ask for the wrong name just to see what happens, but I think they only do this with the root. (Similarly they ask for "index.html" even if you've never used it publicly, and they ask for directory names without final slash.)

It also helps in identifying botnets-- the ones that involve infected browsers, so they request supporting files along with html. If they're from a blocked range, all these requests will come in with the "wrong" form of your domain name as referer, because they never got as far as the rule that would have redirected them. (If there's a 301 from mod_rewrite or mod_alias or even mod_dir, and later a 403 from mod_authz-or-similar, the server only sends out the 403.)

Edit:

ErrorDocument 404 http://example.com/art/404-not-found.html

NOOOOO!
Never, ever use protocol-plus-hostname with an error document. It changes all errors into a 302 redirect. The form is root-relative (or, if you prefer, root-absolute)

ErrorDocument 404 /art/404-not-found.html

Further edit after looking over most recent htaccess, ignoring commented-out rules:

# (1) Block useless bots
...
RewriteRule . - [F]

As written, this rule excludes requests for the root, which come through as ^$ in htaccess (also in any <Directory> section). If this was intentional, leave it. Otherwise change to .?

# (2) Rewrite ...

Here AND EVERYWHERE, change "Rewrite" in comments to "Redirect". You can do this with an unsupervised global replace if you include a space after the word "Rewrite ".

(External) Redirect = send a response to the browser telling them to make a fresh request
(Internal) Rewrite = serve content from somewhere else, but don't tell the browser you're doing it

# (5) Rewrite to HTTPS for secure 'other'.html pages in /bookings2/
...
RewriteRule ^((?:booking-entry|booking-save)\.html)$ [example.com...] [R=301,L]

This is not only an http(s) rule but an URL change, so
(1) OMIT the conditions, because presumably you're redirecting everyone, regardless of protocol
(2) move this rule before #3 to keep all URL-changing redirects together. (Here and elsewhere, when I say "URL" I really mean "path".)

# (10) Rewrite to HTTP for non-secure 'other'.html pages
...
# RewriteRule ^((?:art)/.*)?$

If this rule is to be un-commented, leave out the superfluous inner parentheses. Is this a "ghost" of some earlier, more complicated rule?

# (12) Rewrite to HTTP for non-secure index.html pages
<snip>
# (13) Rewrite to HTTPS for secure index.html pages

These two rules are wrong :( In each case, OMIT all reference to port and/or https, because the point of the rule is to redirect everyone, regardless of port and protocol. The only reason you need two rules is that there are two targets, either http or https. Be sure to test these rules. If they lead to an infinite loop you will need to add a condition that checks whether "index.html" was in the request. Otherwise [NS] alone will do.

Finally rule 14: You may need to split this in two, exactly like the index redirect. One rule for the https directories, one for the others.

[phranque]

This produced the '301' status code correctly. From the 'Live HTTP Headers' result I quoted earlier (#4672968 at 8:30 am on May 21, 2014), it appeared there was a call for authentication. But there was not an authentication pop-up window delivered to the browser; I don't know why.

HTTP/1.1 301 Moved Permanently
...
WWW-Authenticate: Basic realm="Example - Members Only"
...
Location: http://example.com/401.shtml

there must be something wrong with how your HTTP Basic Authentication is configured.
the WWW-Authenticate HTTP Response header should be sent with a 401 response, not a 301.

[Carob]

Having considered all your suggestions and advice - going through all the queries and responses again - and still getting the issues that I have been describing, I began to question whether I need to make the distinction between HTTP and HTTPS responses. I read the contributions of others on this topic, and came to the conclusion that I could avoid most of the problems by simply serving HTTPS pages to all users. Serving HTTPS to all appears not to be a problem.

So I simplified my .htaccess file to:

Options +FollowSymlinks
Options -Indexes

<Files "robots.txt">
Order Allow,Deny
Allow from all
</Files>

Addhandler application/x-httpd-php5 .html .php

ErrorDocument 401 /art/401-unauthorised.html
ErrorDocument 404 /art/404-not-found.html

RewriteEngine on

# (1) Block useless bots
RewriteCond %{HTTP_USER_AGENT} (Baiduspider|YandexBot) [NC]
RewriteRule .? - [F]

# (2) Redirect those seeking superseded pages to their replacements:
# 'als' to 'afa', 'bookings1' to 'bookings2', 'sf1' to 'pf1', 'sf2' to 'pf2', and 'sfa' to 'pf2'
RewriteRule ^als(.*)$ [example.com...] [R=301,L]
RewriteRule ^bookings1(.*)$ [example.com...] [R=301,L]
RewriteRule ^sf([12].*)$ [example.com...] [R=301,L]
RewriteRule ^sfa(.*)$ [example.com...] [R=301,L]

# (3) Redirect requests for index.html to show the path without the file name
RewriteRule ^(.*)index\.html$ [example.com...] [R=301,L]

# (4) Host Name Canonicalisation (use non-www form of secure domain)
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443 [OR]
RewriteCond %{HTTP_HOST} !^(example\.com)?$ [NC]
RewriteRule ^(.*)$ [example.com...] [R=301,L]

The changes are to rules (3) and (4).

However, I have two queries:

(a) If a send a curly request like:

http://www.old-example.com/private

ie. wrong protocol, wrong domain, and requiring authorisation, I am served the page 'https://example.com/art/401-unauthorised.html' straight away. Obviously, the correct protocol and domain have been served. Looking at the Live HTTP Headers, I can see that the need for authorisation has been recognised, but it has not been requested. Even if I previously make an accurately addressed request, and enter username and password, making that wide-of-the-mark request in the same browser session does not make use of the cached authorisation: the 401 error still results.

The 'private' subdirectory has an .htaccess file:

AuthType Basic
AuthName "Private - Members Only"
AuthUserFile /home/example/.htpasswds/.htpasswd
AuthGroupFile /dev/null
Require valid-user

Logically, I would have thought my request would go to the .htaccess in the 'public_html' directory and get the protocol and domain sorted, as it does, and then, armed with an accurate request, encounter the .htaccess in the target directory, request authorisation, and having received it, serve the intended page.

Why isn't this occurring, or is there something I have missed?

(b) By Rule (3):

RewriteRule ^(.*)index\.html$ [example.com...] [R=301,L]

I have intended to simplify the URL the user sees. I would have preferred the '/afa/index.html' page, for example, to be served as ' [example.com...] rather than [example.com...] . I have tried:

RewriteRule ^(.*[^/])/?index\.html$ [example.com...] [R=301,L]

and positive lookahead constructions, but I haven't been able to achieve any better result than Rule (3) achieves. Can you please help me with the syntax?

[lucy24]

Logically, I would have thought my request would go to the .htaccess in the 'public_html' directory

Before anything else: Does the inner htaccess contain the line
RewriteEngine On
without the line
RewriteOptions inherit
? If yes to both, then all earlier RewriteRules will be abandoned as if they had never existed.

I would have preferred the '/afa/index.html' page, for example, to be served as ' [example.com...]
Careful. You're mixing two different processes. Normally, "index.html" only applies to actual, physical directories. But if your code uses URLs like

/afa

then you're talking about extensionless URLs, where

/afa

is quietly rewritten to

/afa.php

(or html or whatever it really is).

Question for people who know search-engine behavior better than I do: If requests for

afa.html

are redirected to

afa

will the search engine ask for

afa/

and hence

afa/index.html

? You only need rules for requests that actually occur.

[edited by: Ocean10000 at 10:50 pm (utc) on Jul 27, 2014]
[edit reason] Fixed Requested Mispelling. [/edit]

[Carob]

Thank you for your reply, Lucy.

Query (a):

Before anything else: Does the inner htaccess contain the line
RewriteEngine On
without the line
RewriteOptions inherit
? If yes to both, then all earlier RewriteRules will be abandoned as if they had never existed.

No, the .htaccess file in the 'private' subdirectory contains only the entries I quoted earlier:

AuthType Basic
AuthName "Private - Members Only"
AuthUserFile /home/example/.htpasswds/.htpasswd
AuthGroupFile /dev/null
Require valid-user

The root .htaccess corrects the wrong protocol, wrong domain, and wrong canonical form, and encounters the inner .htaccess file in the 'private' subdirectory, as shown by the authenticate line in the Live HTTP Headers report:

http://www.old-example.com/private

GET /private HTTP/1.1
Host: www.old-example.com
...

HTTP/1.1 301 Moved Permanently
Date: Mon, 28 Jul 2014 00:43:38 GMT
Server: Apache
WWW-Authenticate: Basic realm="Private - Members Only"
Location: [example.com...]
...
----------------------------------------------------------
[example.com...]

GET /art/401-unauthorised.html HTTP/1.1
Host: example.com
...

HTTP/1.1 200 OK
Date: Mon, 28 Jul 2014 00:43:38 GMT
Server: Apache
...
----------------------------------------------------------
...

Yet the user is not asked for username and password, and is served the 401 error page directly. I don't understand why authentication is not requested, nor taken from cache when it's available, and the error page is served.

Also, with the redirecting having occurred due to the root .htaccess, there is no further redirecting necessary in the 'private' subdirectory, so no need to inherit rules from the root. However, if I change the .htaccess in the 'private' subdirectory to:

AuthType Basic
AuthName "Private - Members Only"
AuthUserFile /home/example/.htpasswds/.htpasswd
AuthGroupFile /dev/null
Require valid-user

RewriteEngine on
RewriteOptions inherit

then I still see the same behaviour: the 401 error page is served immediately with all the redirecting from the root .htaccess having already occurred correctly.

Query (b):
Lucy, I gather you mean that I should leave the redirect:

RewriteRule ^(.*)index\.html$ [example.com...] [R=301,L]

as it is, so that, for example, '/afa/index.html' is redirected to 'https://example.com/afa/' . I was trying to cover the circumstance where one advises a customer to go to the 'afa' page to find an answer, as "example.com/afa", and trying to have the URL in the browser reflect that. You're saying it's wrong to do that?

[Carob]

Later ...
Further research has provided the answer for query (b). The discussion of the DirectorySlash Directive:

http://httpd.apache.org/docs/current/mod/mod_dir.html

shows the issues and possible security pitfalls of deleting trailing slashes for directories.
So I will stick with Rule 3 as written above (msg:4691011 at 7:50am on 27 July).

Only the solution to query (a) behaviour remains. Though the "curly" request seems too extreme to occur in practice, it is possible and even likely given past experience. So fixing the behaviour so that authentication is sought and the correct page is delivered is important.

[Carob]

In the public_html directory the .htaccess file (edited down) includes:

Options +FollowSymlinks
Options -Indexes

ErrorDocument 401 /art/401-unauthorised.html
ErrorDocument 404 /art/404-not-found.html

RewriteEngine On

# Redirect requests for index.html to show the path without the file name
RewriteRule ^(.*)index\.html$ [example.com...] [R=301,L]

# Host Name Canonicalisation (use non-www form of domain)
RewriteCond %{HTTPS} =off [OR]
RewriteCond %{SERVER_PORT} !443 [OR]
RewriteCond %{HTTP_HOST} !^(example\.com)?$ [NC]
RewriteRule ^(.*)$ [example.com...] [R=301,L]

and the .htaccess file in the subdirectory /private has only these entries:

AuthType Basic
AuthName "Private - Members Only"
AuthUserFile /home/example/.htpasswds/.htpasswd
AuthGroupFile /dev/null
Require valid-user

If I request

http://www.oldexample.com/private

the 401 page is served:

https://example.com/art/401-unauthorised.html

Note that the wrong protocol, the wrong domain, and wrong canonical form in the request have all been corrected, and the subdirectory .htaccess file in /private has been encountered as shown in the Live HTTP Headers Report:

http://www.oldexample.com/private

GET /private HTTP/1.1
Host: www.oldexample.com
...

HTTP/1.1 301 Moved Permanently
Date: Mon, 04 Aug 2014 01:41:04 GMT
Server: Apache
WWW-Authenticate: Basic realm="Private - Members Only"
Location: [example.com...]
...
----------------------------------------------------------
[example.com...]

GET /art/401-unauthorised.html HTTP/1.1
Host: example.com
...

HTTP/1.1 200 OK
Date: Mon, 04 Aug 2014 01:41:05 GMT
Server: Apache
...
----------------------------------------------------------
...

Yet authorisation was not requested before serving the error page. The Live HTTP Headers Generator page shows the direct progression to the error page:

#request# GET [oldexample.com...]
GET /private
#request# GET [example.com...]
#redirect# GET /art/401-unauthorised.html
...

Can somebody please explain this behaviour and offer a solution or point me in the right direction?

(A request not containing these errors is met with a request to authorise access, and once satisfied, the page /private/index.html is served as intended. Any request to the wrong domain and requiring authentication causes this error.)

Should I contain the authentication requirements for the subdirectory (and there are other subdirectories similarly affected) within the top .htaccess file by using the <Directory> directive, or is this directive appropriate only to use in the httpd.conf file (to which I do not have access)?

Alternatively, can the authentication specification for the subdirectories be contained within the top .htaccess file by using SetEnvIf ? If so, how? The reading I have done has not helped me to a solution.

Adding:

RewriteEngine On
RewriteOptions Inherit

or

RewriteEngine On
RewriteOptions InheritBefore

to the subdirectory .htaccess file does not seem to fix the behaviour.

Rob