Forward slash afterfilename

[alhermette]

I just noticed in my logs that there are requests where a forward slash and additional directories/parameters have been added. The odd thing is that the server is returning a 200 status code instead of a 404.

I'm not sure that it is an .htaccess problem but here's my current code in there:

ErrorDocument 404 /common/errors/404.php

IndexIgnore *
Options All
Options +FollowSymLinks -Indexes

RewriteEngine On

# Force www
RewriteCond %{HTTP_HOST} ^(example\.com)$ [NC]
RewriteRule ^(.*)$ http://www.%1/$1 [R=301,L]

# Remove index file
RewriteCond %{THE_REQUEST} \ /(.+/)?index\.(php|html)(\?.*)?\ [NC]
RewriteRule ^(.+/)?index\.(php|html)$ /$1 [NC,R=301,L]

Can anyone shed any light as to how I can convince my server to throw a 404 for requests like this:

mydomain.com/index.php/Russia.php?x=blue-widgets/Russia.php

[edited by: incrediBILL at 8:41 pm (utc) on Apr 12, 2012]
[edit reason] exemplified URLs [/edit]

[lucy24]

Sorry, but I gotta digress a little. Both of your illustrative RewriteRules are making the server do more work than it needs to do.

RewriteCond %{HTTP_HOST} ^(example\.com)$ [NC]
RewriteRule ^(.*)$ ht tp://www.%1/$1 [R=301,L]

No capture needed. In fact, to cover all bases you should express it as

RewriteCond %{HTTP_HOST} !(^www\.example\.com$)?
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

:: fingers crossed because I have a long history of mistyping* this Condition ::

That covers http 1.0 (rare but still used by some humans and/or legitimate search engines) as well as the freak requests with appended port like :8080. Anchors in the "pattern" side of the Rule aren't necessary here, because a Regular Expression is greedy by default: it will capture as much as it can unless you give it a specific reason not to.

This rule should be the very last Redirect, because it's only there to pick up requests that haven't already been dealt with by other Redirects. Including...

RewriteCond %{THE_REQUEST} \ /(.+/)?index\.(php|html)(\?.*)?\ [NC]
RewriteRule ^(.+/)?index\.(php|html)$ /$1 [NC,R=301,L] 

Again, you don't need the parts before and after "index.xtn" in the Condition.

RewriteCond %{THE_REQUEST} index\.(php|html) [NC]
RewriteRule ^(([^/.]+/)*)index\.(php|html)$ http://www.example.com/$1 [NC,R=301,L]

The part before "index.xtn" simply isn't needed. And you don't need to anchor the Request because that has already been taken care of in the Rule. What you do need to do is fine-tune the beginning of the Rule so it captures groups of directories but doesn't go too far into the "index.whatever" before stopping. Those nanoseconds add up.

Add the complete protocol-plus-domain in all Redirects.

Does your site actually use both .php and .html? If not, restrict the Rule to the forms you use. Depending on site configuration, it may even be appropriate to make separate Rule-plus-Condition pairs for php and html versions.

Now then. Key question which you forgot to answer: Does your site actually use parameters anywhere? If yes, are they part of visible page names, or only used behind the scenes as the result of a Rewrite?

Next question: A 200 response doesn't happen in a vacuum. Where are these bogus requests going? Remember that your logs don't show rewrites (as opposed to redirects). You can only deduce them from the filesize-- the number after the 200. If you've got a php script that handles all requests, the explanation may lie there rather than in htaccess.

Oh, and probably irrelevant, but: If you've set
Options -Indexes
then what does the
IndexIgnore *
directive do? It seems superfluous.


* In fact I almost forgot to escape the first \. period, so that's my typo for the day.

[g1smd]

Yep, it's a typo.

!(^www\.example\.com$)?

=>

!^(www\.example\.com)?$

Make sure you have AcceptPathInfo off.

[alhermette]

Thanks for the replies guys.

g1smd - !^(www\.example\.com)?$ Thanks for that.

AcceptPathInfo Off sounds like it may be what I need but I amn not sure of the syntax required. I added the following to my .htaccess file but it does not seem to be working as expected:

<Files *.php>
AcceptPathInfo Off
</Files>

Lucy24 - Thanks for your help, I seem to have it mostly working now. As you can probably tell .htaccess is not exactly my strong point. Here's my current .htaccess file in full.

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName example.com
AuthUserFile /home/example/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/example/public_html/_vti_pvt/service.grp

ErrorDocument 404 /common/errors/404.php

Options All
Options +FollowSymLinks -Indexes

<Files *.php>
AcceptPathInfo Off
</Files>

RewriteEngine On

# Force www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

# Remove index file
RewriteCond %{THE_REQUEST} index\.(php|html) [NC]
RewriteRule ^(([^/.]+/)*)index\.(php|html)$ http://www.example.com/$1 [NC,R=301,L]

I am still able to enter www.example.com/index.php/nonsense/rubbish/non-existant.php and I get served my home page. In answer to your questions:

Does your site actually use both .php and .html?

Yes

Does your site actually use parameters anywhere? If yes, are they part of visible page names, or only used behind the scenes as the result of a Rewrite?

Yes it does use query string parameters and they are visible. I have copied my entire .htaccess file above - there are no rewrites concerning query strings at all.

A 200 response doesn't happen in a vacuum. Where are these bogus requests going? Remember that your logs don't show rewrites (as opposed to redirects). You can only deduce them from the filesize-- the number after the 200. If you've got a php script that handles all requests, the explanation may lie there rather than in htaccess.

As far as I can see a request for www.example.com/index.php/nonsense/rubbish/non-existant.php serves the content found at www.example.com/index.php I have no php scripts involved. The server seems to be ignoring everything after the filename whereas I would prefer it to throw a 404.

If you've set
Options -Indexes
then what does the
IndexIgnore *
directive do? It seems superfluous.

I have removed it.

Many thanks for your help.

[lucy24]

As far as I can see a request for www.example.com/index.php/nonsense/rubbish/non-existant.php serves the content found at www.example.com/index.php

Uh-oh. That sounds as if somewhere along the line, everything after "index.php" is getting deleted. You said htaccess, which I assume means you're on shared hosting and there's stuff going on in the config file that you have no control over. But it can't hurt to have a look at your host's control panel-- by whatever name-- and make sure you never accidentally set a preference that tells the server to do the name-chopping thing.

But the rule is written as a Redirect, which means you're not simply serving content; the browser's address bar is changing. Is that what happens when you enter a garbage URL?

To test whether the rule is working at all, add a Condition that limits it to yourself-- for example, by saying "only from such-and-such IP". And change the target from
http://www.example.com/$1

to something like
http://www.example.com/foobar.html

You will of course get your 404 page-- but your browser's address bar will say where it thinks you are. If it says "foobar.html" the rule is working. If it just gives your domain name, with or without "index.whatever", then something else is intercepting the request before it ever reaches the RewriteRule in your .htaccess.

[alhermette]

OK I contacted my hosting company about this because I've never changed any settings anywhere and their support is second to none.

The site is on a Liquidweb VPS. I checked some other sites that I have and also got the support guys to check some of their sites. They all react in an identical manner.

If you append garbage after an html page like this:

www.example.com/directory/page.html/garbage/non-existant.php

it will throw a 404. If you append garbage after a php page like this:

www.example.com/directory/page.php/garbage/non-existant.php

it will serve the contents of page.php and ignore the garbage. The url in the address bar will remain unaltered with all the garbage intact.

Liquidweb's support team say that that is just how it works. It looks like the best way forward then is to create a rewrite in .htaccess to get rid of the garbage. It would need to match any file that ends in .php and remove anything after that unless it is a querystring (because I do use querystrings quite a lot). How would I do that?

Thanks for the help

[lucy24]

By weird coincidence, I have just this second (really) tested what happens if I stick a bogus parameter onto the name of a random html page--which, afaik, shouldn't even have parameters. The page is served up exactly as normal. But further checking confirms that if I continue after the .html -- that is, as if the URL simply wasn't finished yet-- I get the desired 404.

Happily you don't need to make an exception for query strings, because mod_rewrite ignores them by default. That is, it stashes the query in a safe place, executes the rule and reattaches the query unchanged-- unless you have specifically told it to do otherwise.

Now, a normal URL only contains one full stop: the one right before the extension. (Only yesterday there was a question elsewhere in this Forum about filenames with multiple full stops, like .txt.zip. Can we assume you don't have anything like that?)

RewriteRule ^([^.]+\.[^/.]+)[/.] http://www.example.com/$1 [R=301,L]

Beginning anchor is crucial; ending anchor isn't needed. "Capture everything up to the first period (full stop). Capture the full stop. Capture anything after the . that is neither a . nor a directory slash. If you meet another full stop or a directory slash, stop and redirect."

It doesn't matter what, if anything, comes after that first unwanted . or / It only matters that there's something after the extension. If there's nothing-- or if the request doesn't contain an extension in the first place-- the rule will simply not execute.

If the problem is limited to php files-- or some other specific extension-- narrow it down to

RewriteRule ^([^.]+\.php)[/.] http://www.example.com/$1 [R=301,L]

If people start hitting you with garbage extensions like .2az, more fine-tuning is in order. But if you made the server check for everything that could possibly be requested,* you'd never get out of htaccess alive.

:: quick detour here to verify that my own htaccess doesn't have a closing anchor after ".php" (blocked because I don't use them) ::


* No need for a million monkeys. A single cat walking across the keyboard will do nicely.

[alhermette]

Now, a normal URL only contains one full stop: the one right before the extension. (Only yesterday there was a question elsewhere in this Forum about filenames with multiple full stops, like .txt.zip. Can we assume you don't have anything like that?)

The only time that I have extra full stops in the url is if they exist in a querystring so I guess that wouldn't affect anything in .htaccess

Whilst I was playing around with your code I came across some browser specific behaviour that I was unaware of. I started off with a blank .htaccess file just to verify what the default behaviour of the server is. when I entered www.example.com/index.php it ended up on www.example.com. Now that's the exact behaviour that I was after but as I had stripped all of the rewrite rules out of .htaccess I was somewhat surprised - this was using Firefox 11.0.

I then noticed that in IE8 entering the same url resulted in no change in the address bar (it remained as www.example.com/index.php).

Having put the rewrite rules back into .htaccess the index.php was stripped off leaving www.example.com/

I wasn't aware that browsers reacted differently to urls and I assume that what is shown in Firefox's address bar is the result of its own 'cosmetic' rewriting or something. As I am interested in how the search engines see my urls I need to remove the trailing slash as well for consistancy. I've tried a couple of different rewrite rules but to no avail so far. How do I remove a trailing slash from all urls as well?

Here's all my rewrite rules in full from .htaccess:

ErrorDocument 404 /common/errors/404.php

Options All
Options +FollowSymLinks -Indexes

RewriteEngine On

# Force www
RewriteCond %{HTTP_HOST} !^(www\.travel-adaptor\.com)?$
RewriteRule (.*) http://www.travel-adaptor.com/$1 [R=301,L]

# Remove garbage info after .php
RewriteRule ^([^.]+\.php)[/.] http://www.travel-adaptor.com/$1 [R=301,L]

# Remove index file
RewriteCond %{THE_REQUEST} index\.(php|html) [NC]
RewriteRule ^(([^/.]+/)*)index\.(php|html)$ http://www.travel-adaptor.com/$1 [NC,R=301,L]

# Remove trailing slash
RewriteRule ^(.+[^/])/$ http://www.travel-adaptor.com/$1 [R=301,L]

Thanks for all your help (nearly there now!).

[lucy24]

I wasn't aware that browsers reacted differently to urls and I assume that what is shown in Firefox's address bar is the result of its own 'cosmetic' rewriting or something. As I am interested in how the search engines see my urls I need to remove the trailing slash as well for consistancy.

Oh, now that's interesting. One part of what you noticed is done by all browsers, except possibly some very very old one. If you type in (or bookmark)

www.example.com

the browser itself will add a trailing slash before the request ever reaches your server. So you can get rid of "index.php" but not the preceding slash unless you want to risk an infinite redirect. (The kind that your browser intercepts, not the Apache kind.)

Since the "index.php" only gets removed in some browsers, you don't have to worry that the config file itself contains this rewrite. That could get messy.

Your rules are still in the wrong order, but we'll get there. And you really do need to use example.com. You probably did some jiggery-pokery to keep your examples from turning into clickable links-- but if you use example.com you won't have to, because the Forums are coded to exempt (www.)example.(any reasonable extension). And when people cut&paste they don't have to do the same jiggery-pokery all over again.

Now, here you've got the dreaded Leading .+ which is almost as bad as a leading .*

RewriteRule ^(.+[^/])/$ http://www.example.com/$1

The form you need is ^(([^/.]+/)*[^/.]+)/$

where the change from [^/] to [^/.] is not functionally necessary but will shave a few nanoseconds off server time because it can then stop as soon as it meets an extension.

But don't change it yet. Directories have a final slash; pages don't.

/directory/otherdirectory/ = /directory/otherdirectory/index.php

while

/directory/pagename = /directory/pagename.php

(Simplifying for the sake of example.)

The final slash gets added if you want the requested URL to look like a directory. (If it really is a directory you need not do anything, because that is mod_dir's job.) Conversely, it only gets deleted if you want the requested URL to look like a page. If a human comes along and types in

/directory/

you don't want to redirect them to

/directory

because there's no such place. So you'll need some more Conditions.

[alhermette]

OK what you say sounds logical but I am not sure if I am getting closer or more confused now.

One of the things that concerns me is not having more than one version of my homepage accessible for SEO purposes. No matter what someone types in:

example.com
www.example.com
www.example.com/index.php
www.example.com/
example.com/index.php
example.com/

The content of index.php should be served but the address bar show just:

www.example.com

I think that we have achieved that with the exception of Internet Explorer adding a trailing slash. As that's browser specific I don't see that being an SEO issue.

As for the rest we have managed to get rid of garbage from the end of urls quite nicely and we are forcing the use of www so it is just a matter of deciding how to deal with those trailing slashes.

My thoughts are that it should be done as follows:

www.example.com/page.php -> leave it alone
www.example.com/directory/page.php -> leave it alone
www/example.com/directory/index.php -> www.example.com/directory/

Unless I am mistaken it looks like we don't need any code to remove trailing slashes so can I just remove that part?

Changing the order (hopefully correctly?) this is what I have now:

ErrorDocument 404 /common/errors/404.php

Options All
Options +FollowSymLinks -Indexes

RewriteEngine On

# Remove garbage info after .php
RewriteRule ^([^.]+\.php)[/.] http://www.travel-adaptor.com/$1 [R=301,L]

# Remove index file
RewriteCond %{THE_REQUEST} index\.(php|html) [NC]
RewriteRule ^(([^/.]+/)*)index\.(php|html)$ http://www.travel-adaptor.com/$1 [NC,R=301,L]

# Remove trailing slash
#RewriteRule ^(([^/.]+/)*[^/.]+)/$ http://www.travel-adaptor.com/$1 [R=301,L]

# Force www
RewriteCond %{HTTP_HOST} !^(www\.travel-adaptor\.com)?$
RewriteRule (.*) http://www.travel-adaptor.com/$1 [R=301,L]

Are we there yet?

[smallcompany]

Can anyone shed any light as to how I can convince my server to throw a 404 for requests like this:

mydomain.com/index.php/Russia.php?x=blue-widgets/Russia.php

How is your PHP configured? What many call garbage after slash is what PHP uses in certain (or many) situations.
If you do not use that characteristic of PHP, you should turn it off, as per g1smd:
Make sure you have AcceptPathInfo off.

This is a headache that caused me post a lot in both Apache and PHP forums. My problem was that wherever PHP was configured as CGI (on shared hosting plans) I had trouble when calling on PHP (AcceptPathInfo settgins in .htaccess were ignored).
On servers where PHP was configured as Apache, AcceptPathInfo off did the work.

Did you post your whole .htaccess here?

In my case, this happens only if I use AddHandler or AddType directives to parse HTML as PHP.
If I comment them out, I get 404 as I want.
Take a look into your server configuration (you said VPS). IF you have the full control, then httpd.conf is the place to start I believe.

One of my sites lost PR on all of its pages (except home) because of this. When I finally turned AcceptPathInfo off on it, Google ended showing over 70k non-existing pages in WMT. I started serving 410 for all of them as they never really existed, so 301 was not something I liked.
Now, the numbers are slowly going down in WMT.
And it all started with a single link like this:
/page.html/

That slash caused all of the internal links to become like:

page.html/folder/otherpage.html

and it went into perpetual mode with no end until I stopped it.

In short, if you have the full control over the server, stop poking around with .htaccess and configure the server in the way so you get 404 on such requests while you have the server working in the way you want.
Then, see what kind of requests you're getting from search engines like Google, plus see if there is still an external link with trailing slash.
Then, do the fix in .htaccess (if needed) to stop 404 requests if you cannot influence the external link.

[alhermette]

Make sure you have AcceptPathInfo off

I accept that this does sound like the solution but I have no idea how or where to set this.

[smallcompany]

AcceptPathInfo off

You put this line into your .htaccess.

When having the full control over the server, there may be a better way of doing it, I'm not sure as I use shared hosting only.

In any case, put it into your .htaccess and see if you still get 200.

[alhermette]

Thanks smallcompany. Unfortunately it made no difference when I put that in (and commented the appropriate rewrite rule). The garbage information stayed in the address bar and the page gave a 200 response. Back to the rewrite rules I suppose.

This is everything in my .htaccess now. Can anyone see any problems with it?

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName example.com
AuthUserFile /home/example/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/example/public_html/_vti_pvt/service.grp

ErrorDocument 404 /common/errors/404.php

Options All
Options +FollowSymLinks -Indexes

RewriteEngine On

# Remove garbage info after .php
RewriteRule ^([^.]+\.php)[/.] http://www.example.com/$1 [R=301,L]

# Remove index file
RewriteCond %{THE_REQUEST} index\.(php|html) [NC]
RewriteRule ^(([^/.]+/)*)index\.(php|html)$ http://www.example.com/$1 [NC,R=301,L]

# Force www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

Thanks for the help

[smallcompany]

Put this into a file with PHP extension (like phpinfo.php):

<?php
echo phpinfo();
?>

Call it from your browser: example.com/phpinfo.php

What does it say for Server API?

[alhermette]

It says CGI/FastCGI

[smallcompany]

It says CGI/FastCGI

You said it's VPS. Who configured the server?

[alhermette]

It's a standard Liquidweb configuration. I requested a few extras to be included in the build that are needed by some applications. other than that it's a default setup.

[g1smd]

Check the Apache manual for the exact AcceptPathInfo syntax.

The content of index.php should be served but the address bar show just:

www.example.com

No. The correct URL is

www.example.com/

with a trailing slash.

Trailing slash denotes "folder". This also applies to "root".

[smallcompany]

AcceptPathInfo Off

Ask the support why this directive in .htaccess gets ignored.

[lucy24]

Ask the support why this directive in .htaccess gets ignored.

Ask them how it gets ignored. If I'm reading the right details in the right order, it's covered under the same AllowOverride declaration as, well, just about everything you'd ever want to do in htaccess, including most of mod_setenvif and mod_rewrite. So it can't get ignored unless a whole bunch of other stuff also gets ignored-- and you'd notice!

Check the Apache manual for the exact AcceptPathInfo syntax.

Hm, right, on rare occasions you'll meet someone whose htaccess file is hair-splittingly case-sensitive. There was once a post from someone who had to say either "RewriteEngine on" or "RewriteEngine On"-- don't remember which, only that the other casing didn't work.

Apache's three options [httpd.apache.org] are On Off Default. Title Case.

The treatment of requests with trailing pathname information is determined by the handler responsible for the request. The core handler for normal files defaults to rejecting PATH_INFO requests. Handlers that serve scripts, such as cgi-script and isapi-handler, generally accept PATH_INFO by default.

Hm, I think this explains a small head-scratcher that came up in another thread recently.

Let's not even consider the possibility that your Apache installation is older than 2.0.30, the earliest version that allows this directive.

[smallcompany]

Any word from support?