Welcome to WebmasterWorld Guest from 18.207.132.114

Forum Moderators: Ocean10000 & phranque

Message Too Old, No Replies

Trying to 301 redirect via our .htaccess file

     
9:47 am on Jan 4, 2019 (gmt 0)

New User

joined:Jan 4, 2019
posts: 3
votes: 0


I am trying to add a straighforward redirect to the .htaccess file of the website i'm (supposedly) administering.

Sadly I don't really understand the way this website has been set up due to experience in front end HTML/CSS work mostly, so i am trying to tiptoe around the core functionality where possible so that nothing breaks.

Having come to the issue of redirecting existing pages to new pages on the updated site that we are going to replace the old one with in february, I am just trying to test 301 redirects in the existing .htaccess file so that i can confirm that these work. But nothing seems to.

I've tried several versions which i've posted below, but none of these appears to have any effect?

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^business-mobile.htm /blahtest.htm [L,R=301]
RewriteRule ^(.*/)?\.svn/ - [F,L]
ErrorDocument 403 "Access Forbidden"
RewriteCond %{HTTP_USER_AGENT} libwww-perl.*
RewriteRule .* – [F,L]
</IfModule>


Or -

<IfModule mod_rewrite.c>
RewriteEngine On
RedirectPermanent business-mobile.htm blahtest.htm
RewriteRule ^(.*/)?\.svn/ - [F,L]
ErrorDocument 403 "Access Forbidden"
RewriteCond %{HTTP_USER_AGENT} libwww-perl.*
RewriteRule .* – [F,L]
</IfModule>


Or -

<IfModule mod_rewrite.c>
RewriteEngine On
RedirectPermanent business-mobile.htm blahtest.htm
RewriteRule ^(.*/)?\.svn/ - [F,L]
ErrorDocument 403 "Access Forbidden"
RewriteCond %{HTTP_USER_AGENT} libwww-perl.*
RewriteRule .* – [F,L]
</IfModule>


From what I understand, the lines
RewriteRule ^(.*/)?\.svn/ - [F,L] ErrorDocument 403 "Access Forbidden"
should come out as we do not use svn any longer (this is quite an old file / website) but other than that, I am quite far out of my depth here. I'd really appreciate any advice from anyone familiar with how an htaccess file functions?
10:29 am on Jan 4, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


welcome to WebmasterWorld [webmasterworld.com], stripeycat!

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^business-mobile.htm /blahtest.htm [L,R=301]
RewriteRule ^(.*/)?\.svn/ - [F,L]
ErrorDocument 403 "Access Forbidden"
RewriteCond %{HTTP_USER_AGENT} libwww-perl.*
RewriteRule .* – [F,L]
</IfModule>


this looks closest to what you probably want.

a few things that caught my attention:

- the <IfModule mod_rewrite.c> envelope is unnecessary.
you already know mod_rewrite is installed, so no need to check for it.

- if you are using mod_rewrite anywhere, you must use mod_rewrite everywhere.
that means forgoing any redirects with mod_alias directives (i.e. RedirectPermanent) and using an equivalent RewriteRule directive, as in this version.

- the [F] flag send a 403 response, so the last 4 directives specify the custom 403 error message and send 403 responses to any requests for paths ending in ".svn/" or any requests by user agents identifying as libwww-perl.

- if a ".svn/" path no linger exists you can remove that rule and the response will be a 404 instead of a 403.

- if your RewriteRule is a 301 redirect you usually want to specify the full canonical protocol and hostname in the substitution string:
RewriteRule ^business-mobile.htm$ https://www.example.com/blahtest.htm [L,R=301]


what response did you get for the RewriteRule version when https://www.example.com/business-mobile.htm is requested?
11:50 am on Jan 4, 2019 (gmt 0)

New User

joined:Jan 4, 2019
posts: 3
votes: 0


Hi Phranque, many thanks for coming back to me about this!

I've gone through each part of your feedback just below -

- the <IfModule mod_rewrite.c> envelope is unnecessary.
you already know mod_rewrite is installed, so no need to check for it.


Is there a way for me to check that this 'mod_rewrite' is definitely installed? Unfortunately i don't know that this is the case.

i've tried to create a file called 'info.php' in the root directory of the website, as this is apparently a way to show which modules are loaded or installed, but this doesnt seem to work? i can't locate the text 'mod_rewrite' anywhere in the resulting page.

- if you are using mod_rewrite anywhere, you must use mod_rewrite everywhere.
that means forgoing any redirects with mod_alias directives (i.e. RedirectPermanent) and using an equivalent RewriteRule directive, as in this version.


I have to be completely honest, i dont understand this :/ that module 'mod-rewrite' only appears the once in this file from what i can see, at <IfModule mod_rewrite.c>; it's not shown anywhere else?

- the [F] flag send a 403 response, so the last 4 directives specify the custom 403 error message and send 403 responses to any requests for paths ending in ".svn/" or any requests by user agents identifying as libwww-perl.

- if a ".svn/" path no linger exists you can remove that rule and the response will be a 404 instead of a 403.


I do sort of understand this... for now i think i'm going to leave the content there at the moment until i'm more confident in modifying this document!

- if your RewriteRule is a 301 redirect you usually want to specify the full canonical protocol and hostname in the substitution string:

RewriteRule ^business-mobile.htm$ https://www.example.com/blahtest.htm [L,R=301]


i've updated what i had in the original to match what you have here, so the full code now looks like:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^business-mobile.htm$ https://www.example.com/blahtest.htm [L,R=301]
RewriteRule ^(.*/)?\.svn/ - [F,L]
ErrorDocument 403 "Access Forbidden"
RewriteCond %{HTTP_USER_AGENT} libwww-perl.*
RewriteRule .* – [F,L]
</IfModule>


(The URL in the above example seems to be truncated when i preview it, but in my code it has the full 'https:www...' listed in the actual code)

what response did you get for the RewriteRule version when https://www.example.com/business-mobile.htm is requested?


Unfortunately this new code still doesnt seem to work as the response is to load the same business-mobile.htm URL, is it possible that this 'mod_rewrite' is not on or enabled on the apache server that they are running here, and that this could be stopping the rewrite from working?

[edited by: phranque at 11:57 am (utc) on Jan 4, 2019]
[edit reason] Please Use example.com [webmasterworld.com] [/edit]

12:14 pm on Jan 4, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


Is there a way for me to check that this 'mod_rewrite' is definitely installed? Unfortunately i don't know that this is the case.

i had assumed you were already using mod_rewrite.

if you have sufficient access to the server, you can use the following command (or an equivalent) to list the installed apache modules:
apachctl -M

source:
[httpd.apache.org...]

i've tried to create a file called 'info.php' in the root directory of the website, as this is apparently a way to show which modules are loaded or installed, but this doesnt seem to work?

this will help you with the php modules installed, but depending on how php is running on your server (e.g. mod_php vs cgi), it may not show apache modules.
7:00 pm on Jan 4, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


Why is your ErrorDocument directive in the middle of the <IfModule> envelope? ErrorDocument is a core directive, so it can't possibly have anything to do with the availability or nonavailability of mod_rewrite or, for that matter, any other module. All it means is that if for some reason you don't have mod_rewrite, this ErrorDocument directive will not be recognized. Worse, it creates confusion for you, the site administrator, because you'll go looking for the directive and won't find it.

If you do not have use of mod_rewrite, change hosts. The only other thing that would prevent a RewriteRule from executing is if you have additional RewriteRules in a deeper directory along the same file path, without an inheritance option. (This is rare.)

Simple check:
Make a rule that says
RewriteRule foobar http://example.com/widget.html [R=302,L]
replacing "http://example.com" with your own protocol and sitename. Now open your browser and request example.com/foobar (or anything in your site, so long as the URL includes the string "foobar"). Did you get redirected to widget.html? You will, of course, see your site's ordinary 404 page, but the browser address bar will say widget.html

There is a bit of boilerplate somewhere hereabouts on how to clean up an htaccess file. (In fact it is near at hand because I've got it on my hard drive, but don't want to post the same thing again and again if it can be found on the site.)

:: wandering off to look ::

Try this thread [webmasterworld.com]. It's the most recent one I could find. The boilerplate is about halfway down the thread.
7:30 pm on Jan 4, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


I am trying to add a straighforward redirect to the .htaccess file of the website

That does not require mod_rewrite:

Redirect 301 /business-mobile.htm https://www.example.com/blahtest.htm

Neither does specifing a custom Error document:

ErrorDocument 404 /errors/404.htm

You might "rewrite" your code like this:


# Redirect old page
Redirect 301 /business-mobile.htm https://www.example.com/blahtest.htm
# Custom errors
ErrorDocument 404 /errors/404.htm
# Invoke mod_rewrite
RewriteEngine On
# Identify undesirable UA
RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC]
# Block access
RewriteRule /*$ – [F]

But I am not going to test it for you.

...
8:15 pm on Jan 4, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


Do not mix mod_alias redirects with mod_rewrite redirects.
9:17 pm on Jan 4, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


You might "rewrite" your code like this:

you shouldn't.
11:54 pm on Jan 4, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


Do not mix mod_alias redirects with mod_rewrite redirects.

I didn't mix them, they are entirely separate (the Rewrite directives will run first).

When not to use mod_rewrite

mod_rewrite should be considered a last resort, when other alternatives are found wanting. Using it when there are simpler alternatives leads to configurations which are confusing, fragile, and hard to maintain. Understanding what other alternatives are available is a very important step towards mod_rewrite mastery.

Simple Redirection

mod_alias provides the Redirect and RedirectMatch directives, which provide a means to redirect one URL to another. This kind of simple redirection of one URL, or a class of URLs, to somewhere else, should be accomplished using these directives rather than RewriteRule.

when there are Redirect and RewriteRule directives in the same scope, the RewriteRule directives will run first, regardless of the order of appearance in the configuration file

Source: [httpd.apache.org...]

...
1:57 am on Jan 5, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


Word for the day: cherry-picking.
The most common situation in which mod_rewrite is the right tool is when the very best solution requires access to the server configuration files, and you don't have that access. Some configuration directives are only available in the server configuration file. So if you are in a hosting situation where you only have .htaccess files to work with, you may need to resort to mod_rewrite.
...
The use of RewriteRule to perform this task may be appropriate if there are other RewriteRule directives in the same scope. This is because, when there are Redirect and RewriteRule directives in the same scope, the RewriteRule directives will run first, regardless of the order of appearance in the configuration file.

In the case of the http-to-https redirection, the use of RewriteRule would be appropriate if you don't have access to the main server configuration file, and are obliged to perform this task in a .htaccess file instead.
Et cetera et cetera. Source: the same page as above.
3:42 am on Jan 5, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


I didn't mix them, they are entirely separate (the Rewrite directives will run first).

assuming there is or eventually will be a hostname canonicalization redirect (which will necessarily use mod_rewrite), you are potentially creating an unnecessary chained redirect as follows:
- when http://example.com/business-mobile.htm is requested, the mod_rewrite directives will fire first, as you pointed out, and the hostname canonicalization ruleset will redirect the request to https://www.example.com/business-mobile.htm
- when https://www.example.com/business-mobile.htm is subsequently requested it will fire the mod_alias rule and provide a second redirect to https://www.example.com/blahtest.htm
- now the visitor is waiting on the third request...

this ultimate request can be achieved with the first redirect if you do everything using mod_rewrite.
10:57 am on Jan 5, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


Word for the day: cherry-picking

Here is what you wrote in full:

Do not mix mod_alias redirects with mod_rewrite redirects.

Can you supply an authoritative citation for that shouty command?

It is perfectly acceptable - as anticipated in the Apache documentation - to use mod_alias and mod_rewrite directives in tandem (as I often do, with no problems whatsoever).

What you shouldn't do is mix code from both modules in a doomed attempt to affect the sequence of execution.

assuming there is or eventually will be a hostname canonicalization redirect (which will necessarily use mod_rewrite), you are potentially creating an unnecessary chained redirect

As I demonstrated in a previous [webmasterworld.com...] post, that is not true.

My .htaccess coding style has been used on dozens of websites on different servers for fifteen years without issue, and is fully compliant with both the Apache documentation and the wisdom of the former moderator of this forum.

Code ignores urban myth.

...
12:49 pm on Jan 5, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


assuming there is or eventually will be a hostname canonicalization redirect (which will necessarily use mod_rewrite), you are potentially creating an unnecessary chained redirect

... that is not true


i just tested the equivalent to the following on apache 2.4:
Redirect 301 /test https://www.example.com/blahtest

RewriteEngine On

RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$ [NC,OR]
RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://www.example.com/$1 [R=301,L]

when i request /test from a non-canonical hostname it redirects twice before it requests https://www.example.com/blahtest

then i tested the equivalent to the following on apache 2.4:
RewriteEngine On

RewriteRule ^test$ https://www.example.com/blahtest [R=301,L]

RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$ [NC,OR]
RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://www.example.com/$1 [R=301,L]

when i request /test from a non-canonical hostname the first redirect is to https://www.example.com/blahtest

q.e.d.
4:39 pm on Jan 5, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


The boilerplate is about halfway down the thread.

That is not an "authoritative source" and it needs correcting:

Step 4a: Get rid of mod_alias. If your htaccess file contains any mod_rewrite directives, it can't use mod_alias (Redirect... by that name), or things may happen in the wrong order.

Things will definitely happen in the "right" order.

It may not be the order you hoped for, but it will be the "right" one.

Apache is designed to use mod_alias and mod_rewrite in tandem - when both are invoked the mod_rewrite directives are executed first, then the mod_alias directives are executed using the results from the mod_rewrite module as the starting point.

There is therefore no "unnecessary chained redirect" unless there is some other problem with your code.

And as the Apache documentation clearly states:

Understanding what other alternatives are available is a very important step towards mod_rewrite mastery... mod_rewrite should be considered a last resort, when other alternatives are found wanting

The idea that you can't (or shouldn't) use mod_alias and mod_rewrite in the same .htaccess file is apparently an online urban myth derived from a misunderstanding of old WebmasterWorld posts.

It seems irresponsible to propagate it further.

...
4:44 pm on Jan 5, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


q.e.d.

I'm afraid not, phranque.

Here is the full (exemplified) .htaccess file I used to test just now:

# Redirect old page
Redirect 301 /test.htm https://www.example.com/test/test.htm

# Custom errors
ErrorDocument 404 /errors/404.htm

# Invoke mod_rewrite
RewriteEngine On

# Canonical & Encryption
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^www\.example\.com [NC,OR]
RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://www.example.com/$1 [R=301,L]

# Identify undesirable UAs
RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^-$

# Block access
RewriteRule /*$ - [F]

This is the request I typed into my browser:

example.com/test.htm

These are the full results shown in the log files:

[05/Jan/2019:16:24:48 +0000] "GET /test.htm HTTP/1.1" 301 617 "-"
[05/Jan/2019:16:24:49 +0000] "GET /test/test.htm HTTP/1.1" 200 38 "-"

The first line is from the http log, the second from the https log.

One redirect, no chain.

Code ignores urban myths.

...
9:40 pm on Jan 5, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


I am trying to add a straightforward redirect to the .htaccess file of the website i'm (supposedly) administering.

Welcome to WebmasterWorld, stripeycat.

Anyone can try my code posted above.

It is fully compliant with the Apache documentation and recommendations.

The code has worked flawlessly on every shared hosting platform I have ever used, Apache versions 1.3 to 2.4 inclusive.

All you need to do is change the file and domain names.

I have never had a problem using mod_rewrite and mod_alias in the same file.

Anyone can try my code posted above.

...
9:51 pm on Jan 5, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


These are the full results shown in the log files:

it must be magic.
or perhaps there is a log or log entry somewhere that you missed.
what was the value of the Location header in the first 301 response?
1:47 am on Jan 6, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


it must be magic.
It's a kind of magic that doesn't happen on my own site, if so. In an excess of paranoia--even though I know I have tested this before, since that's kinda the whole point of having a test site*--I tried it again, using my tried-and-true foobar-to-widget pattern. (The 302 is because** the RedirectMatch rule was put in for testing and therefore didn't say anything about Permanent or 301.)

http:
{my IP} - - [05/Jan/2019:17:26:54 -0800] "GET /foobar HTTP/1.1" 301 569 "-" "{my UA}"

https:
{my IP} - - [05/Jan/2019:17:26:54 -0800] "GET /foobar HTTP/1.1" 302 734 "-" "{my UA}"
{my IP} - - [05/Jan/2019:17:26:55 -0800] "GET /widget.html HTTP/1.1" 404 1178 "-" "{my UA}"

Why did the second redirect (mod_alias redirect to https://www.example.com/widget.html) take up fully 165 more bytes than the first redirect (mod_rewrite redirect to https://www.example.com/foobar)? Aw, heck, who knows.

Apache docs are intended primarily for server administrators. (And are, apparently, written by people who hate Regular Expressions.) htaccess has a whole set of its own issues, but it simply isn't a major part of what Apache deals with.


* That is: not only to try things that might create a 500 error without affecting any "real" site, not only so I can say that I'm speaking from direct personal experience, but because a test site has tiny logs that can conveniently be pulled many times a day. As I speak, this site's http logs are less than 1k, the https logs not much bigger.
** Also because I forgot, making this a case of doing the right thing for the wrong reason.
4:10 am on Jan 6, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


Why did the second redirect (mod_alias redirect to https://www.example.com/widget.html) take up fully 165 more bytes than the first redirect (mod_rewrite redirect to https://www.example.com/foobar)? Aw, heck, who knows.

the difference is probably the caused by the "SSL"-related response headers (i.e the typical difference in headers between a HTTP response and a HTTPS response)
6:13 am on Jan 6, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


Oh, thanks. Didn't know that.

For an encore, can you speculate about the wild variation in response sizes to robots.txt requests on an http site?

:: quick detour to confirm that I remembered this right ::

It ranges all over the map, from Cliqzbot and a few others at 1029k, through Semrushbot at 1048k, assorted others including bing, google and Seznam at 1085k--and then it jumps up to 2775 for DotBot and 2831 for Yandex and MJ12. (Also a few mere 294k, but those are requests getting a special stripped-down robots.txt because reasons.) Poring over headers suggests that the big size difference is because some accept gzip or similar encoding and some don't--but what on earth causes the small differences? (This is one of those things like balancing a checkbook, where two cents unaccounted for is more worrying than $200.)
8:44 pm on Jan 6, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


it must be magic

No, I'm pretty sure it is called Apache.

I have been using that coding style for about 15 years without any issues (as have others).

I learned it from jdMorgan, formerly of this parish.

It has nothing to do with SSL as I only started using that in 2017.

I currently use the code on sites spread over two standard commercial shared hosting servers.

I have used it on at least half a dozen others in the past.

I am aware that Apache configurations can differ slightly between shared hosting companies.

But I have yet to come across one that this code did not work on.

Apache docs are intended primarily for server administrators

I could have sworn that this part was written especially for you:

Understanding what other alternatives are available is a very important step towards mod_rewrite mastery.

Your understanding of mod_alias appears wanting.

Things definitely happen in the right order.

...
10:32 am on Jan 7, 2019 (gmt 0)

New User

joined:Jan 4, 2019
posts: 3
votes: 0


Hello again.

First, thanks to everyone here who has offered input around my problem.

Although i still do not understand the bulk of what each of you is covering in your posts (as this is fairly far beyond my capabilities as a graphic designer!), I really do appreciate the feedback.

Lucy, i have attempted an add of the line
RewriteRule foobar http://example.com/widget.html [R=302,L]
to the existing htaccess file here; unfortunately, this does not seem to work on the website that is currently live. I do not see the text that you've outlined in the URL.

Samizdata, I have saved out the existing htaccess file and uploaded the content you'd posted earlier -


# Redirect old page
Redirect 301 /business-mobile.htm https://www.myurl.com/blahtest.htm
# Custom errors
ErrorDocument 404 /404.htm
# Invoke mod_rewrite
RewriteEngine On
# Canonical & Encryption
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^www\.myurl\.com [NC,OR]
RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://www.myurl.com/$1 [R=301,L]
# Identify undesirable UAs
RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^-$
# Block access
RewriteRule /*$ - [F]


But unfortunately this redirect does not work either.

Would it be of use if I were to post the htacess file in its entirety here? The total line count is only 28 lines, and my concern is that i may only be providing a part of a picture here instead of describing the file in its entirety. I do not have the knowledge around how each part of this document functions alongside the others to make an informed decision, sadly. Also I'm very wary of rushing in to make changes as have been warned about doing so in the past. I have read around each line within the file to try and make sense of what each does and also how they relate to each other, but with limited success!
2:37 pm on Jan 7, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5507
votes: 5


stripey,
There's nothing wrong with making changes in your htaccess, however you must always follow two simple rules:
1) keep a backup copy (just rename the old working online file with extra characters at files end (i. e. .htaccess19_0107)
2) always make sure to test your website after uploading a newer version of htaccess.
6:56 pm on Jan 7, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


I do not see the text that you've outlined in the URL.
Uh-oh, this calls for more investigating. If you've added the experimental line or, for that matter, any similar experimental line--replacing example.com with your real sitename, of course--and request foobar.html, where do you end up? Matter of fact, it doesn't actually matter if you forgot to make the change, because then you'd end up at the real example dot com seeing their 404 screen instead of yours.

Things definitely happen in the right order.
If you define “right order” as “two separate redirects” then there is nothing more to be said.
2:32 am on Jan 9, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


there is nothing more to be said

In the early days of this forum, it was common practice to use mod_alias and mod_rewrite in the same .htaccess file (we all used Apache 1.3 back then).

It was then realised that some hosting providers ran these modules in a different order that could cause a problem, so the advice was changed.

It first became "don't use mod_access and mod_rewrite in the same file if you intend to move hosts", as you might get a problematic one and be forced to change your code (actually no big deal for a few Redirects).

Sometime later the potential canonical issue that phranque has raised became apparent - it would certainly be an issue on a "problematic" hosting setup, but as I wasn't on one I had no need to change my ways (and none of the other hosts I have used since then have been "problematic" either).

It was all about the unpredictability of the module order.

In Apache 2.4 the module order is predictable, mod_rewrite before mod_alias.

It is the "right" order.

Understanding that is a very important step towards mod_rewrite mastery.

Note that mod_rewrite tries to guess whether you have specified a file-system path or a URL-path by checking to see if the first segment of the path exists at the root of the file-system. For example, if you specify a Substitution string of /www/file.html, then this will be treated as a URL-path unless a directory named www exists at the root or your file-system (or, in the case of using rewrites in a .htaccess file, relative to your document root), in which case it will be treated as a file-system path. If you wish other URL-mapping directives (such as Alias) to be applied to the resulting URL-path, use the [PT] flag as described below.

Perhaps your problematic RewriteRules need a [PT] flag.

It apparently works like this:

passthrough|PT Forces the resulting URI to be passed back to the URL mapping engine for processing of other URI-to-filename translators, such as Alias or Redirect.

I would say that the Apache documentation not only clearly states that mod_rewrite and mod_alias can be used together, it tells you how to use .htaccess to force optimal compatibility of the Apache configuration.

[httpd.apache.org...]

In theory, my code above should therefore be revised to:

# Redirect old page
Redirect 301 /test.htm https://www.example.com/test/test.htm

# Invoke mod_rewrite
RewriteEngine On

# Canonical & Encryption
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^www\.example\.com [NC,OR]
RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://www.example.com/$1 [R=301,PT]

I haven't tested it.

But I am in no hurry to tell my hosts they configured Apache incorrectly.

...
3:49 am on Jan 9, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15944
votes: 890


Perhaps your problematic RewriteRules need a [PT] flag.
This thread is about .htaccess. In fact, the word appears in the thread title. The [PT] flag is not used in htaccess, because, quote,
the PT flag is implied in per-directory contexts such as <Directory> sections or in .htaccess files


Interestingly, the PT flag is not simply ignored (the way a superfluous L would be ignored in locutions like [F,L]); it creates a 400 error. Not 400-class, but numerical 400 “Bad Request”. Unfortunately, this type of error doesn't translate into an error-log entry giving further information.
5:28 am on Jan 9, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


One redirect, no chain.

Things definitely happen in the right order.

In theory, my code above should therefore be revised to:

I haven't tested it.

i did.
(without the [PT] flag to avoid the 400 obviously)
2 redirects.
11:56 am on Jan 9, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


The [PT] flag is not used in htaccess

Quote incomplete:
The only way to circumvent that is to rewrite to -.

And the point you wilfully miss is that Apache 2.4 is built to use mod_rewrite and mod_alias in sequence by design.

The use of the [PT] flag causes it to be treated as a URI instead. That is to say, the use of the [PT] flag causes the result of the RewriteRule to be passed back through URL mapping, so that location-based mappings, such as Alias, Redirect, or ScriptAlias, for example, might have a chance to take effect.

This design is intended to deal with any redirect chain issue caused by RewriteRules.

My hosts appear to deal with it at a higher level, as I never get the chain.

That also means I can't test the relevant code as the issue is already dealt with on my shared hosting servers.

My advice is to RTFM and keep testing your RewriteRule until you get it right.

Get rid of mod_alias. If your htaccess file contains any mod_rewrite directives, it can't use mod_alias (Redirect... by that name), or things may happen in the wrong order.

As I said before, it seems irresponsible to propagate that nonsense further.

There was an issue with module order under Apache 1.3 - but that has been resolved in later versions.

Nowadays, things definitely happen in the right order.

...
1:20 am on Jan 10, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11875
votes: 246


And the point you wilfully miss is that Apache 2.4 is built to use mod_rewrite and mod_alias in sequence by design.

The use of the [PT] flag ...

show of hands:
- how many of you reading this have access to the server config files?

Nowadays, things definitely happen in the right order.

... on your host.

i'd like to point out to anyone reading this far that most of samizdata's claims or suggestions in this thread have fallen into one or more of the following categories:
- not actually tested
- is irrelevant to the subject (i.e. doesn't apply to .htaccess context as specified in the thread title)
- is a nonstandard feature that is specific to samizdata's web hosting service

do your own testing.
3:43 am on Jan 11, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1378
votes: 18


it must be magic

Apologies phranque, I fed you a bum steer earlier.

My host's cPanel identifies Apache 2.4.37 as the server version.

But it seems that the webserver under the hood is actually LiteSpeed, apparently a commercial Apache fork optimised for shared hosting companies (with .htaccess support being one of the attractions).

Compatibility is such that my numerous 1500+ line .htaccess files have always worked perfectly without me ever suspecting that I was no longer on Apache. Embarrassing for me, certainly, but technically rather interesting.

LiteSpeed apparently adds an extra server variable and invisibly deals with the redirect chain issue, making the use of mod_alias and mod_rewrite together nice and easy for shared hosting customers.

As it damn well ought to be.

The log snippets I posted earlier are accurate - viewing live headers confirms that there is only one redirect and no chain, a result similar to the advertised effect of the [PT] Passthrough flag in Apache.

So the boilerplate advice these days might now be:

Avoid using mod_alias and mod_rewrite together unless you either have access to the Apache configuration above Directory level or are on a LiteSpeed server (where the two modules seem to play nice).

The current Apache .htaccess implementation omits the Passthrough option offered at VirtualHost level, so a rewritten URI cannot be passed internally to mod_alias before execution (as it sometimes needs to be) in a shared hosting environment.

This may lead to a chain of two 301 Permanent redirects being executed, generally undesirable and not popular with search engines.

A future Apache release needs to address this issue - mod_rewrite and mod_alias will play nice if you ask them, but you can't ask them from a shared hosting account.

Apologies again for the confusion.

I have no problem using mod_rewrite and mod_alias in the same .htaccess file, and no problem with other people using "mod_rewrite for everything".

My shared hosting provider supports both approaches, life is peachy.

But posts concerning LiteSpeed are extremely rare on WebmasterWorld.

A previous moderator felt they might not belong in this forum.

And he usually knew what he was talking about.

...
This 32 message thread spans 2 pages: 32