.htaccess mod rewrite and http authentication problem

.htaccess code to add "www" and "/" to URLs and allow http authentication

[smmairaj]

Hi,

I have a task that I am unable to get to work for a few days and I am literally pulling my hair now. Any help would be really really appreciated! Here is my case:

I have to achieve the following three things:

1. redirect non www traffic to www urls
Ex: http://example.com/ should redirect to http://www.example.com/

2. add a slash to the end of the urls
Ex: http://www.example.com/some-file/parameters should redirect to http://www.example.com/some-file/parameters/

3. Basic http authentication - I have a directory under root ("/" or public_html), say it's called "dev". I want this to have a basic http authentication access. (done using "Password Protect Directories" option in the "CPanel" which probably writes to apache's config file)

Well, sounds pretty simple and I am very well able to achieve these goals individually but not all at the same time. Following is something similar that my .htaccess looks like right now:

Options +FollowSymLinks 
RewriteEngine on 

RewriteCond %{HTTP_HOST} !^www\.example\.com$ 
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,QSA,L] 

RewriteCond %{REQUEST_FILENAME} !-f 
RewriteCond %{REQUEST_URI} !(.*)/$ 
RewriteRule ^(.*)$ http://www.example.com/$1/ [L,R=301] 

RewriteRule ^contact-us/?$ /contact-us.php 

RewriteRule ^products/?$ /products.php 
RewriteRule ^products/([a-zA-Z0-9-]+)/([a-zA-Z0-9-]+)/?$ products.php?catid=$1&productid=$2 
RewriteRule ^products/([a-zA-Z0-9-]+)/?$ products.php?catid=$1 

Options -MultiViews 
DirectoryIndex index index.php index.html 
ErrorDocument 404 /404.shtml 

The problem with the current .htaccess is that i am unable to access any of the directories (I have indexing disabled on the directories) and it simply gives a 404 error when i try to open them in a browser.

Ex: when I try to access http://www.example.com/images/ it gives me a 404

When I look at the http headers it tells me that the server first responds with a "301 Moved Permanently" status and in the "Location:" bit of this response I see http://www.example.com/401.shtml. Then in response for a request for 401.shtml, the server throws a 404 error. However, if I am not using the above .htaccess then I get the normal 403 Forbidden page for these directories.

The current .htaccess behaves similarly (throws a 404 error) with my dev directory (which is a basic authentication required directory) as well and here's where it bothers me.

I have extensively googled for a solution but to no use. I found a similar problem mentioned on this forum here:

[webmasterworld.com...]

and when I searched for "RewriteOptions inherit" I get these:

[webmasterworld.com...]
[webmasterworld.com...]

but many things go way above my head and probably do not address what exactly I am looking for.

I think I should mention that I am new to .htaccess coding. However, with so much research and time spent on getting this to work, I now totally understand every bit of my current .htaccess, although, the deeper concepts are still vague to me.

Please help me!

And yes, before I close I would like to mention that another important concern I have is that I want Multiviews to be disabled. I have observed that keeping Multiviews enabled restricts googlebot to index my pages.

Thanks a lot in advance for your time and help!

Cheers

[smmairaj]

Anybody on this :(

[phranque]

welcome to WebmasterWorld [webmasterworld.com], smmairaj!

it seems as if that DirectoryIndex directive should do it for you.
have you checked your server logs for clues?

[g1smd]

Each rule should have [L] on the end.

Additionally ^(.*)$ can be simplified to (.*) too.

It is generally not a good idea to add a slash to the end of a URL unless that URL is a real folder. That goes against the HTTP specifications. Extensionless URLs do not end with a slash.

[smmairaj]

Thanks phranque and g1smd for you replies.

phranque, I couldn't figure out the relation of DirectoryIndex Directive to my problem :( However, I got what I wanted!

Thanks g1smd for you advices, I actually implemented all of them. Here is how the modified part of my .htaccess looks like now:

I replaced:

RewriteCond %{REQUEST_FILENAME} !-f  
RewriteCond %{REQUEST_URI} !(.*)/$  
RewriteRule ^(.*)$ http://www.example.com/$1/ [L,R=301] 

with

RewriteCond %{REQUEST_FILENAME} !-d 
RewriteRule ^(.*)/$ [%{HTTP_HOST}...]  [R=301,L] 

I was trying to add a trailing slash to the end of the urls so that the pages are served only from a single url pattern. Since you have said that it's against the HTTP specs, I did the reverse of it. Now the slashes are removed from the end of the urls if it's not a directory. And removing the previous bit of code also solves my problem.

I also removed the ^ and $ from the following code:

RewriteCond %{HTTP_HOST} !^www\.example\.com$ 
RewriteRule (.*) http://www.example.com/$1 [R=301,QSA,L] 

and also added [L] to the end of each rule.

2 questions:

1. Does removing ^ and $ has some performance effect?

2. Is it necessary to have [L] at the end of each rule, although it does not have a condition preceding it? For ex:

RewriteRule ^products/?$ /products.php [L] 

The above line has no condition preceding it in my code. Or is it so that if I put an [L] at the end then the parsing of the code is stopped once that rule is met? And if there's no [L], then it goes on to parse the following lines?

Thanks again! You guys have been really helpful!

Editing... One more question, is adding a .html / htm / php or whatever extention to end of the urls advisable?

[edited by: eelixduppy at 8:46 am (utc) on Nov. 27, 2008]
[edit reason] exemplified [/edit]

[jdMorgan]

1. Performance is slightly improved because the mod_rewrite parser has two less characters to process. Since a ".*" pattern is 'greedy', it will match as much as possible. Therefore, there is no need to use "^" and "$" when the pattern consists only of ".*" or "(.*)".

2. If the [L] flag is omitted, then all rules will be processed, even if an earlier rule has been invoked and has changed the request-URI. This is wasteful of processor resopurces, and can also lead to unexpected results, because the request-URI has been changed.

You don't need to use [QSA] unless you wish to append new query string data to an existing query string. If your RewriteRule does not specify a new query string, then the original query string will be passed through the rule unchanged.

If your server is accessible via HTTP/1.0 (that is, if it is on a unique, unshared IP address), then you should make provisions for the HTTP_HOST to be blank, and do not redirect it if it is. That changes your RewriteCond pattern to:

RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$

Otherwise, an HTTP/1.0 client would cause your server to go into an infinite redirection loop, because HTTP/1.0 clients do not send the HTTP Host header, HTTP_HOST will be blank, and you would repeatedly attempt to redirect the request because it would match the "NOT www.example.com" pattern.

Jim