Problems with rewrite in .htaccess

Redirection is to an unknown path

[LeeU]

I have set-up an .htaccess file (listed below) for a newly designed website (the domain has been up for years). One of the errors I am receiving is when a visitor goes to the URL

http://example.com/Newsletters/5765-2005.pdf

they should be directed to

http://example.com/newsletters/5765-2005.pdf

(small "n" in "newsletters"). But they somehow are redirected to

http://example.com/articles/letters/5765-2005.pdf

The sub-directory "letters" does not exist, nor is it listed in the .htaccess file, unless it is shortening the "newsletters" directory and placing the "articles" directory in front of it (which I don't know how it would be doing that).

I have checked the page and it says it's a 301 permanent move and gives the correct directory where it has moved, i.e.,

http://example.com/newsletters/5765-2005.pdf

Any one have any ideas of what is going on?

.htaccess file

AddHandler application/x-httpd-php54s .php
AddHandler server-parsed .html
AddHandler server-parsed .htm

Options +SymLinksIfOwnerMatch
RewriteEngine on
RewriteBase /
ServerSignature Off
Options ExecCGI Includes IncludesNOEXEC SymLinksIfOwnerMatch -Indexes

# backward compatibility ruleset for rewriting document.htm to document.php
RewriteRule ^(.*)\.htm$  $1 [C,E=WasHTML:yes]
RewriteCond %{REQUEST_FILENAME}.php -f
RewriteRule ^(.*)$ $1.php   [S=1,R]
RewriteCond %{ENV:WasHTML}  ^yes$
RewriteRule ^(.*)$ $1.htm

# Blocks access from libwww-perl user-agents and URLS which include the command "=http:"
RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)=http [NC]
RewriteRule ^(.*)$ - [F,L]

# Redirect www to non-www
RewriteCond %{HTTP_HOST} ^www.example.com$ [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]

# Prevent hotlinking of images
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?example\.org/.*$ [NC]
RewriteRule \.(gif|jpg|swf|flv|png)$ /feed/ [R=302,L]

# Used for the errorpage.php file
ErrorDocument 400 /errorpage.php
ErrorDocument 401 /errorpage.php
ErrorDocument 403 /errorpage.php
ErrorDocument 404 /errorpage.php
ErrorDocument 500 /errorpage.php

# Caching
# ExpiresActive On
# ExpiresDefault A0
<ifModule mod_headers.c>
# 1 YEAR
<filesMatch ".(ico|pdf|flv)$">
Header set Cache-Control "max-age=29030400, public"
</filesMatch>
# 1 WEEK
<filesMatch ".(jpg|jpeg|png|gif|swf|css|js)$">
Header set Cache-Control "max-age=604800, public"
</filesMatch>
# 2 DAYS
<filesMatch ".(xml|txt)$">
Header set Cache-Control "max-age=172800, proxy-revalidate"
</filesMatch>
# 1 MIN
<filesMatch ".(html|htm|php)$">
Header set Cache-Control "max-age=60, private, proxy-revalidate"
</filesMatch>
</ifModule>
<ifModule mod_expires.c>
# any Expires Directives go here
</ifModule>
# End Caching

# compress text, html, javascript, css, xml:
FilterDeclare COMPRESS
FilterProvider COMPRESS DEFLATE resp=Content-Type $text/html
FilterProvider COMPRESS DEFLATE resp=Content-Type $text/css
FilterProvider COMPRESS DEFLATE resp=Content-Type $text/javascript
FilterChain COMPRESS
FilterProtocol COMPRESS DEFLATE change=yes;byteranges=no
# <IfModule mod_deflate.c>
# AddOutputFilterByType DEFLATE text/plain
# AddOutputFilterByType DEFLATE text/html
# AddOutputFilterByType DEFLATE text/xml
# AddOutputFilterByType DEFLATE text/css
# AddOutputFilterByType DEFLATE application/xml
# AddOutputFilterByType DEFLATE application/xhtml+xml
# AddOutputFilterByType DEFLATE application/rss+xml
# AddOutputFilterByType DEFLATE application/javascript
# AddOutputFilterByType DEFLATE application/x-javascript
# </IfModule>

# <Files 403.shtml>
# order allow,deny
# allow from all
# </Files>

<Files php.ini>
Order allow,deny
Deny from all
</Files>

RewriteRule ^Articles/?(.*)$ http://example.com/articles/$1 [R=301,L]
RewriteRule ^Events/?(.*)$ http://example.com/events/$1 [R=301,L]
RewriteRule ^News/?(.*)$ http://example.com/articles/$1 [R=301,L]
RewriteRule ^Newsletters/?(.*)$ http://example.com/newsletters/$1 [R=301,L]

[lucy24]

Options +SymLinksIfOwnerMatch
<snip>
Options ExecCGI Includes IncludesNOEXEC SymLinksIfOwnerMatch -Indexes

Stop right there.

Edit:
Oh, all right, I continued looking.

RewriteRule ^News/?(.*)$ http://example.com/articles/$1 [R=301,L]
RewriteRule ^Newsletters/?(.*)$ http://example.com/newsletters/$1 [R=301,L]

As cosgan dot de would say:
/images/smilie/froehlich/a065.gif

Further edit:
I understand the intention, but it can't be done that way. The directory slash is only optional if there is nothing after it. So the pattern is

blahblah(/.*)?$

without capture, or

blahblah(?:/(.*))?$

with capture. This only works in RegEx engines that don't object to potentially undefined back-references. mod_rewrite is fine with it. Using ?: in the outer group lets you stick with $1 in the target; otherwise it would become $2.

[LeeU]

If I change the line

RewriteRule ^Newsletters/?(.*)$ http://example.com/newsletters/$1 [R=301,L]

to

RewriteRule ^Newsletters(/.*)?$ http://example.com/newsletters/$1 [R=301,L]

I still get the error.

I just saw your addition. I will try that.

[LeeU]

Nope, still doesn't work with:

RewriteRule ^Newsletters(?:/(.*))?$ http://example.com/newsletters/$1 [R=301,L]

[LeeU]

Interestingly, the others work. Only the that doesn't is for the newsletters directory. Would it matter if it directed to PDF files to the directory? (Just trying to make some sense.)

[lucy24]

The problem is not in Newsletters. The problem is in the immediately preceding line, News.

[aakk9999]

The sub-directory "letters" does not exist, nor is it listed in the .htaccess file,

Remove the word "News" from the word "Newsletters" and you get "letters".

The problem is not in Newsletters. The problem is in the immediately preceding line, News.

Exactly!

@LeeU, swap the order of rules to list your Newsletters redirect before your News redirect.

[LeeU]

That's it! Thanks, @aakk9999! It works just fine now.

For my own information, why did it change it the way it did, in the order I had in the original listing?

[aakk9999]

Because you need to put rules in the order of more specific to less specific. Your URL with "Newsletters" was executing "News" rule, because "News" is subset of "Newsletters"

^News/?(.*)$ says "Anything starting with "News" or "News/"

The (.*) after the News in URLs with "Newsletters" then captured the pattern "letters" and this is waht got inserted after the /articles/ (referred to by $1).

When you swapped the lines around, URLs with "Newsletters" matched the ^Newsletters rule first and this one got executed.

[LeeU]

Ah, that makes sense now. Thanks again.

(Now, how do I marked this "solved"?)

[lucy24]

No such button :)

But that was the point of fine-tuning to
(?:/(.*))?$
With this configuration, the rules will work regardless of order, even if one pattern name is contained within another. It also eliminates a problem in the original form, where a filename might be captured with or without leading directory slash.

[aakk9999]

So changing the News rule would allow any order of rules and it is probably a safer solution to use:

RewriteRule ^News(?:/(.*))?$ http://example.com/articles/$1 [R=301,L]

Thank you Lucy for explaining :)

[LeeU]

Ah, now I get it. Yes, thanks for the explanation. It makes sense now. And it works just fine (someday I'll learn regular expression properly).

Thanks to both of you!

[g1smd]

You're not quite done. There's a number of other issues to fix up. These include:
- rewriterules that block access or prevent hot-linking should be listed first
- jpg|jpeg| simplifies to jpe?g|
- [F,L] flag should be [F]
- redirects should be listed before rewrites
- the www to non-www redirect should be the last of the redirects
- the RegEx pattern ^www.example.com$ [NC] should be just !^(example\.com)?$
- in any rewrite with rule target beginning $1, the rule target must be prefixed with a slash
- missing escaping in multiple RegEx patterns.

You might also usefully list the "deny from" rules at the very top of the file.

[LeeU]

Thanks, g1smd. I cleaned it up with those tips. It seems there are many articles about the .htaccess file but I have never seen one that explains the order things should be done (although it always seemed like there should be one).

[LeeU]

Okay, well after doing a site search on Google, I have found I have problems with another domain ("example.com") and one of its sub-domains ("example2.com"). I have listed below excerpts from the .htaccess files (you can see that I have attempted to fix this problem before but, evidently, not very effectively).

One problem I can see is the rewrite to the "blosxom.cgi" file. I need to differentiate between example.com and example2.com domain/sub-domain.

I would appreciate some more help, if possible.

MAIN DOMAIN - excerpted portion

RewriteEngine on

Redirect 301 /emet/weekly.xml http://example2.com/
Redirect 301 /rss-info.htm http://example2.com/rss-info.php
Redirect 301 /articles//going/feedback.php http://www.example.com/going/feedback.php
Redirect 301 /cgi-bin/2me_4see_comment.cgi http://www.example.com/contact.php
Redirect 301 /emet/analysis/arab_claim_to_palestine.htm http://example2.com/analysis/arab_claim_to_palestine.php
Redirect 301 /articles/important-jeru.php http://www.example.com/articles/what-is-so-important-about-jerusalem.php
Redirect 301 /articles/important-jeru.htm http://www.example.com/articles/what-is-so-important-about-jerusalem.php
Redirect 301 /articles/histjeru.php http://www.example.com/articles/a-biblical-history-of-jerusalem.php
Redirect 301 /articles/histjeru.htm http://www.example.com/articles/a-biblical-history-of-jerusalem.php
Redirect 301 /articles/jerucov.php http://www.example.com/articles/jerusalem-a-separate-covenant.php
Redirect 301 /articles/jerucov.htm http://www.example.com/articles/jerusalem-a-separate-covenant.php
Redirect 301 /articles/seventhmonth.php http://www.example.com/articles/index.php
Redirect 301 /articles/touch_not_mine_anointed.php http://www.example.com/articles/index.php
Redirect 301 /articles/camel-needle.php http://www.example.com/articles/index.php
Redirect 301 /articles/camel-needle.htm http://www.example.com/articles/index.php
Redirect 301 /bookReviews/review-paul_the_jewish_theologian.php http://www.example.com/books.php
Redirect 301 /articles/spiritual-authority-what-it-is-and-what-it-isnt.php http://www.example.com/articles/index.php
Redirect 301 /articles/godsplan.php http://www.example.com/articles/index.php
Redirect 301 /articles/blessings.php http://www.example.com/articles/index.php
Redirect 301 /articles/fathers_love.php http://www.example.com/articles/index.php
# Redirect 301 /book_reviews/ http://www.example.com/books.php
Redirect 301 /articles/harry_potter.php http://www.example.com/articles/index.php
Redirect 301 /articles/tradmen.php http://www.example.com/articles/index.php
Redirect 301 /articles/pastors.php http://www.example.com/articles/index.php
Redirect 301 /articles/covering.php http://www.example.com/articles/index.php
Redirect 301 /articles/bride_ready.php http://www.example.com/articles/index.php

RewriteRule ^weblog/?(.*)$ /cgi-bin/blosxom.cgi/$1

RewriteRule ^(.+)\.htm$ http://www.example.com/$1.php [R=301,L]

RewriteCond %{HTTP_HOST} ^example\.org [OR]
RewriteCond %{HTTP_HOST} ^69\.89\.23\.241
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

RewriteRule ^views/?(.*)$ http://example2.com/views/$1 [R=301,L]
RewriteRule ^analysis/?(.*)$ http://example2.com/analysis/$1 [R=301,L]
RewriteRule ^documents/?(.*)$ http://example2.com/documents/$1 [R=301,L]
RewriteRule ^palestinian-myths/?(.*)$ http://example2.com/palestinian-myths/$1 [R=301,L]
RewriteRule ^media/?(.*)$ http://example2.com/media/$1 [R=301,L]

==========================

SUB-DOMAIN - excerpted portion

RewriteEngine On

Redirect 301 /articles/ http://www.example.org/articles/histjeru.php
Redirect 301 /analysis/gaza_and_katrina.php http://example2.com/analysis/gaza-and-katrina.php
Redirect 301 /analysis/palestine-history.pdf http://example2.com/analysis/palestine_history.pdf
Redirect 301 /documents/unsc_resolution_338.php http://example2.com/documents/unsc-resolution-338.php
Redirect 301 /analysis/the_myth_of_occupied_territories.php http://example2.com/analysis/the-myth-of-occupied-territories.php
Redirect 301 /analysis/palestine_history.php http://example2.com/analysis/palestine-history.php
Redirect 301 /gaza-assistance.php http://example2.com/gaza-commentaries-booklets-pictures.php
Redirect 301 /analysis/the_israeli_humanitarian_lifeline_to_gaza.php http://example2.com/analysis/the-israeli-humanitarian-lifeline-to-gaza.php
Redirect 301 /analysis/jewish_blood_who_is_really_behind_the_gaza_flotilla.php http://example2.com/analysis/jewish-blood-who-is-really-behind-the-gaza-flotilla.php
Redirect 301 /analysis/is_israel_bound_by_international_law_to_supply_utilities_goods_and_services_to_gaza.php http://example2.com/analysis/is-israel-bound-by-international-law-to-supply-utilities-goods-and-services-to-gaza.php
Redirect 301 /contact.php http://example2.com/contact/contact.php
Redirect 301 /Blog.cgi/index.rss http://example2.com/cgi-bin/blosxom.cgi/index.rss
Redirect 301 /blog.cgi/index.rss http://example2.com/cgi-bin/blosxom.cgi/index.rss
Redirect 301 /weblog/why-must-an-arab-palestinian-state-be-jew-free.html) http://www.example2.com/weblog/why-must-an-arab-palestinian-state-be-jew-free.html
Redirect 301 /current.htm http://example2.com/
Redirect 301 /current.php http://example2.com/
Redirect 301 /weekly.xml http://example2.com/

RewriteRule ^weblog/?(.*)$ /cgi-bin/blosxom.cgi/$1

RewriteCond %{HTTP_HOST} ^www.example2.com$ [NC]
RewriteRule ^(.*)$ http://example2.com/$1 [R=301,L]

RewriteCond %{HTTP_HOST} example\.org$
RewriteRule ^(.*)$ "http\:\/\/example2\.org\/" [R=301,L]

RewriteRule ^(.+)\.htm$ http://example2.com/$1.php [R=301,L]

RedirectMatch 301 ^/palestinian_myths/$ http://example2.com/palestinian-myths/

RewriteRule ^archives/?(.*)$ http://example2.com/cgi-bin/mail.cgi/archive/emet
RewriteRule ^palestinian_myths/(.*)$ http://example2.com/palestinian-myths/$1 [R=301,L]

Thanks in advance for any help you can give.

[lucy24]

GET RID OF all rules involving mod_alias (Redirect by that name). You may be able to combine mod_alias and mod_rewrite in the config file, but don't try it in htaccess.

Open a copy of your htaccess file in a text editor that does Regular Expressions and run the following global replaces:

# change . to \.
^(Redirect \d\d\d \S+?[^\\])\.
TO
$1\\.
# now change Redirect to Rewrite
^Redirect(?:Match)? 301 /(.+)
TO
RewriteRule $1 [R=301,L]
# and (if necessary)
^Redirect(?:Match)? 410 /(.+)
TO
RewriteRule $1 - [G]

Also get rid of all escaped : colons and / directory slashes AND all quotation marks in targets.

[LeeU]

Thanks! I'll give it a try and get back to you.