htaccess problem - What am I missing?!

htaccess problem

[and1c]

ok guys

I have a vbulletin forum that is quite large.

I recently did an upgrade on it to the latest version and all went well, however I have been penalised by Google PR wise as it seems I have duplicate content...

Looking into it I found my main htaccess is not working properly...

RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.example.co.uk$
RewriteRule ^(.*)$ http://www.example.co.uk/$1 [R=301]
RewriteCond %{HTTP_HOST} ^(www.)?example.co.uk$
RewriteCond %{REQUEST_URI} !^/forum/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /forum/$1
RewriteCond %{HTTP_HOST} ^(www.)?example.co.uk$
RewriteRule ^(/)?$ forum/index.php [L]

ok, basically, everything should redirect to example.co.uk as I now have a mod added that does re writes and this is installed in the /forum/ directory.

So if someone types

http://example.co.uk or
http://www.example.co.uk/forum or
http://www.example.co.uk/index.php

it should automatically redirect to my main URL

http://www.example.co.uk

its not doing this and all three version above give the same content?!

What am I missing please?

[edited by: jdMorgan at 11:01 pm (utc) on Oct. 24, 2008]
[edit reason] example.co.uk [/edit]

[jdMorgan]

You might try adding the Options directive, just in case your host has turned it off at the main server level. Then I'd suggest quite a few clean-ups and tweaks, while we're here:

Options +FollowSymLinks
RewriteEngine on
#
# Externally redirect all non-canonical hostname requests to canonical domain
RewriteCond %{HTTP_HOST} [b]!^(www\.example\.co\.uk)?$[/b]
RewriteRule [b](.*)[/b] http://www.example.co.uk/$1 [R=301[b],L][/b]
#
# Internally rewrite URLs which do not resolve to an existing file or
# directory to the /forum directory, unless this has already been done.
RewriteCond %{REQUEST_URI} !^/forum/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule [b](.*)[/b] /forum/$1 [b][L][/b]
#
# Rewrite requests for root home page to /forum/index.php
RewriteRule [b]^$[/b] /forum/index.php [L]

Having redirected all non-canonical hostname requests in the first rule, it is not necessary to check the hostname on every subsequent rule.

Literal periods in regular-expressions patterns should be escaped. You should allow for a blank hostname in case you get a true HTTP/1.0 connection to this server. If you don't allow for it, then such a connection would result in an 'infinite' redirection loop.

Your redirect should have an [L] flag on it, to avoid exposing the /forum path to clients (e.g. SE 'bots).

Your second rule should have an [L] flag as well, because if it matches, then processing the last rule is a waste of time. (Basically, always use an [L] flag on every rule, unless you have a good reason not to.)

Since ".*" is greedy and will match as much as possible, "(.*)" is equivalent to "^(.*)$, but shorter.

If "example.co.uk/" is requested from your server, RewriteRule will see a blank URL-path, so it is not necessary to use "(/?)" in the final rule. It may actually be counter-productive, because the only way RewriteRule would ever see that slash is if "example.co.uk//" were requested -- two slashes in row. And in that case, you should detect that, and redirect to "example.co.uk/" (one slash) to avoid dup-content problems.

There are several other canonicalization problems you might want to preclude. One of them is a direct client request for /index.php or /index.whatever. Those requests should be externally redirected to "/".

Another one (in this case) would be a direct client request for /forums/<whatever>. These requests should be *redirected* to /whatever, and then your second rule will *internally rewrite* them back to /forum/<whatever>. The difference is that without that new redirect, /forums/<whatever> can be indexed by the SEs, making it a duplicate of /<whatever>.

However, I strongly suggest you hold off on adding these or any new features until you've got your original features working again. When ready, the code for this function has been posted here many times -- Look for the "Redirect index to '/' (or root)" threads, implement that, and then duplicate it and modify it for the "/forums/" problem.

I also suggest always including meaningful comments in your code. This is helpful when posting on forums for assistance, and also to remind you just exactly what you originally had in mind when you revisit the code after several months/years.

Be sure to completely flush your browser cache before testing any changes to your .htaccess file.

Jim

[and1c]

Thanks for your help Jim.

That has been very useful..but I now find myself slightly lost :)

ok, the sites structure is (top down).

site root
forum
**forum subfolders**

I have an add on 'mod' that handles the internal forum URL rewrites. This is in php modules and a few other files, however the 'forum' directory uses the htaccess file below (not my work)

RewriteEngine On

RewriteRule ^(.*)t([0-9]+)(.*)newpost(.*)$ showthread.php?t=$2&goto=newpost [QSA,L]
RewriteRule ^(.*)f(.*)/(.*)t([0-9]+)(.*)/post([0-9]+)(.*)$ showpost.php?p=$6 [QSA,L]
RewriteRule ^(.*)f(.*)/(.*)t([0-9]+)(.*)/([a-z]+)([0-9]+)(.*)$ showthread.php?t=$4&page=$7 [QSA,L]
RewriteRule ^(.*)f(.*)/(.*)t([0-9]+)(.*)$ showthread.php?t=$4 [QSA,L]
RewriteRule ^(.*)f([0-9]+)/([a-z]+)([0-9]+)(.*)$ forumdisplay.php?f=$2&page=$4 [QSA,L]
RewriteRule ^(.*)f([0-9]+)(.*)$ forumdisplay.php?f=$2 [QSA,L]
RewriteRule ^((urllist¦sitemap).*\.(xml¦txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

ok, back to the top level of the site...the site root and the htaccess file that is giving me the trouble!

I am the server admin (centos) and have setup apache. I have not turned off options, only
- indexes
but I added the symlinks code to see and it didnt change it.

however as this htaccess was done ages ago (I cant even remember how long!) the needs of it have now changed.

This is what Im trying to acheieve.

So that all of the following are redirected to

http://www.example.co.uk

****************************************************
http://www.example.co.uk/index.php
http://example.co.uk
http://www.example.co.uk/forum/
http://example.co.uk/forum
****************************************************

How can I best acheieve this without messing up the other URL rewrites>!

thanks

[jdMorgan]

I'm not sure why you're lost. The code I posted simply corrects a few minor problems, but is otherwise identical in function (if somewhat faster to execute) as what you had.

I'll be happy to get into more detail on your added requirement to redirect index files, but not until we get the basic function restored.

The /forum/.htaccess file has nothing to do with domain/hostname redirection, so let's leave that for now as well.

The question is, what is and isn't working? If it isn't working, we'll need specifics. For example, if you type http://example.co.uk/ into your browser (after completely flushing its cache), then what happens?

Jim

[jdMorgan]

Added: Maybe I'm missing something... Is it the case that this is *new* .htaccess code and it is not working, or is it the case that this is *not* new code, and it has stopped working since your upgrade?

Jim

[and1c]

Jim

This code is not working since the upgrade!

The reason I am confused is because only 'parts' of it seem to be working! Although I know several parts of the htaccess are wrong to satisfy what I am now wanting! (I USED to want it to redirect only to /forum/ instead of now being ./.))

ok
*************************************************

http://www.example.co.uk/index.php goes to itself

http://example.co.uk gives a 'cannot find server' error. No code numnber shown

http://www.example.co.uk/forum/ goes to itself

http://example.co.uk/forum goes to itself

additionally , http://example.co.uk/index.html goes to itself and I would like that also to redirect to the site root
****************************************************

Also,I have completely flushed my browsers cache between each 'test' just so we know there are no red herrings!

Does this help us/you out at all?!

thanks

[jdMorgan]

These problems do not make sense. For one thing, example.co.uk giving a 'cannot find server' error is a DNS problem -- Edit your DNS zone file to add an "A" record for "example.co.uk." and note that the trailing period after ".uk" is required.

The index.php and /forum URLs resolving to themselves is happening because your code does not contain any logic to redirect these URLs. Unless you deleted the rules or modified the server configuration file, that cannot be explained.

If this is a local test site, and you do not have DNS set up for it, then add an entry for "example.com.uk" to your 'hosts' file. The location of the hosts file is different on different operating systems, but it is always named 'hosts' -- all-lowercase, and no file extension. The 'hosts' file is your computer's 'local' DNS server. If you later register the domain and point it to a different server, then delete your hosts file entries for that domain.

You should insert the following three new rules before the other rules in the code I posted above to redirect the index.xyz and /forum URLs. Leave everything else as-is, except for changing "example" to your actual domain name, and fixing the pipe characters as noted below:

# Externally redirect direct client requests only for /forum/index.xyy files to /
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/index\.(php¦s?html?) [NC]
RewriteRule ^forum/index\.(php¦s?html?)$ http://www.example.co.uk/ [NC,R=301,L]
#
# Externally redirect direct client requests only for <any_directory_level>/index.xyz
# files to <any_directory_level>/
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.(php¦s?html?) [NC]
RewriteRule ^(([^/]+/)*)index\.(php¦s?html?) http://www.example.co.uk/$1 [NC,R=301,L]
#
# Externally redirect direct client requests only for /forum/<any_file> to /<any_file>/
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/ [NC]
RewriteRule ^forum/(.*)$ http://www.example.co.uk/$1 [NC,R=301,L]

The RewriteCond lines are required to prevent recursion in .htaccess.

Note that all index files, such as
index.php
index.php4
index.php5
index.htm
index.shtm
index.shtml
index.html -or even-
INDEX.HTML
will now be redirected to "/"

In addition, an index file located in /foo/bar/index.php will be redirected to /foo/bar/

Important: Replace all broken pipe "¦" characters above with solid pipe charactes before use; Posting on this forum modifies the pipe characters.

Jim

[edit] Corrected as noted in following post. [/edit]

[edited by: jdMorgan at 3:05 am (utc) on Oct. 25, 2008]

[g1smd]

jd, minor typo, the code doesn't cater for php4 or php5 as printed.

[and1c]

Thanks JD :)

it is a live site and DNS is paid for and hosted with DynDNS
I have all the records setup correctly as I checked! however I recleaned my browser cache (yet again!) and tried it

http://example.co.uk and it worked! redirecting to http://www.example.co.uk

Your changes have fixed a lot of the Dupe content issues I have been having so thanks again..

The only things that are not redirecting and still showing dupe content are

http://www.example.co.uk/forum/

shows the same content as

http://www.example.co.uk

So it seems that

# Externally redirect direct client requests only for /forum/index.xyy files to /
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/index\.(php¦s?html?) [NC]
RewriteRule ^forum/index\.(php¦s?html?)$ http://www.example.co.uk/ [NC,R=301,L]

is not working?

If I am correct in my thinking it should redirect

http://www.example.co.uk/forum/
or http://www.example.co.uk/forum/index.php (or any other index.**)

All to http://www.example.co.uk/

At present the code is working and redirecting /forum/index.php back to /forum/
however it is not going back to site root

A weird one is:

http://example.co.uk/forum/

doesnt redirect anywhere. However

http://example.co.uk/forum/index.php

redirects to http://www.example.co.uk/forum/index.php

!

Any thoughts?

[edited by: jdMorgan at 2:33 pm (utc) on Oct. 25, 2008]

[and1c]

g1smd... can you elaborate please?!?

I corrected the pipe characters

[jdMorgan]

The rule that handles your problem case as you described above is:

# Externally redirect direct client requests only for /forum/<any_file> to /<any_file>/
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/ [NC]
RewriteRule ^forum/(.*)$ http://www.example.co.uk/$1 [NC,R=301,L] 

but I see nothing wrong with the code. It should handle redirecting /forum/ to "/" or /forum/foo to /foo just fine.

Jim

[edited by: jdMorgan at 3:28 am (utc) on Oct. 25, 2008]

[and1c]

Ooops!
My mistake, sorry it was late when i posted that.

# Externally redirect direct client requests only for /forum/<any_file> to /<any_file>/
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/ [NC]
RewriteRule ^forum/(.*)$ http://www.example.co.uk/$1 [NC,R=301,L]

indeed, it seems to be correct syntax wise after reading up some more on htaccess stuff.

I will look into this and post back. Hopefully with a solution or some idea of one! thanks

[and1c]

ok guys.
Ive still got a big problem with this and Im not entirely sure of the best way to tackle it..

The reason being.

http://www.example.co.uk

and

http://www.example.co.uk/forum/
and
http://example.co.uk/forum/

all serve the same content. Meaning I haev been penalised by Google and lost my Pr6 accidentally...

Very annoying!
And The whole forum URL structure (its using a Vb URL rewrite mod for internal URLs)is say
http://www.example.co.uk/forum/f21

Option?:

Change the htaccess which is as shown below at present and redirect instead of to root, to root/forum
but will this cause me other problems?

I have tried several times to get the ruleset working as I need, but I cant quite get it right.

has anyone got any suggestions on what to do please?

Options +FollowSymLinks
RewriteEngine on
#

# Externally redirect direct client requests only for /forum/index.xyy files to /
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/index\.(php¦s?html?) [NC]
RewriteRule ^forum/index\.(php¦s?html?)$ http://www.example.co.uk/ [NC,R=301,L]
#
# Externally redirect direct client requests only for <any_directory_level>/index.xyz
# files to <any_directory_level>/
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.(php¦s?html?) [NC]
RewriteRule ^(([^/]+/)*)index\.(php¦s?html?)$ http://www.example.co.uk/$1 [NC,R=301,L]
#
# Externally redirect all non-canonical hostname requests to canonical domain
RewriteCond %{HTTP_HOST} !^(www\.example\.co\.uk)?$
RewriteRule (.*) http://www.example.co.uk/$1 [R=301,L]
#
# Internally rewrite URLs which do not resolve to an existing file or
# directory to the /forum directory, unless this has already been done.
RewriteCond %{REQUEST_URI} !^/forum/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule (.*) /forum/$1 [L]
# Rewrite requests for root home page to /forum/index.php
RewriteRule ^$ /forum/index.php [L]]

[and1c]

As JD mentioned earlier, I believe the ruleset dealing with this is

# Externally redirect direct client requests only for /forum/<any_file> to /<any_file>/
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /forum/ [NC]
RewriteRule ^forum/(.*)$ http://www.example.co.uk/$1 [NC,R=301,L]

I have tried multiple things with it but cant seem to get it working>?!

Could it be the TFSeo mod that I am running on Vbulletin? I cant see how but I am out of ideas :(

[jdMorgan]

Please be very specific about what you mean by "not working." We can't tell from here if you are getting a 404, a server error, or if blue smoke comes out of your computer when you invoke the code...

Jim

[and1c]

JD

The htaccess is not working as I expected/hoped.

The reason for this is because my site is still serving duplicate content.

The URLS

http://www.example.co.uk

and

http://www.example.co.uk/forum/
and
http://example.co.uk/forum/

All serve identical content. I was wanting the bottom two examples to redirect to http://www.example.co.uk

however I dont want all URLs
eg/ http://www.example.co.uk/forum/F24/topic1

to redirect for obvious reasons (eg/ the forum will break)

Only the specific ones

http://www.example.co.uk/forum
and
http://example.co.uk/forum

I have chopped and changed the htaccess to try and factor this in and I cannot get

http://www.example.co.uk/forum to redirect to the main site URL :(

Is this any better explained?!