301 redirect best practice

[kellyman]

Hi Guys Implemented a 301 a few months ago (14 weeks to be precise), i have rankings that have dived and not recovered, The original 301 was implemnted by PHP but a few weeks ago i have asked my developer to change this to htaccess which now 50% is done via the old site attempting to do the new site once we confirmed all is best practice

Scenario is

www.example.net was renamed to www.example.com

Both sites sit on different servers along with different I.P's

the new sites 301 is currently handled by php and although we get all the correct headers and such i believe there to be an issue as rankings are still 80% down, this php handles the canonicals and removes the forward slash pretty simple

As i mentioned we are going to implement a htaccess version on both servers

On the old we are going to add the below which is a simple redirect

RewriteEngine on
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

we were advised to add the below, but we got an internal 500 server error however the single line code works just fine, however there must be a reason for the below and is it best practice...

RewriteEngine On
RewriteCond %{HTTP_HOST} ^example.net$ [OR]
RewriteCond %{HTTP_HOST} ^www.example.net$
RewriteRule (.*)$ http://www.example.com/$1 [R=301,L]


On the new server which is currently being handle by PHP I have sought help already a month or 2 ago but wanted to check that what i was doing was correct and no unwanted chaining being forced, so not implemented yet it will be in the next few days once i get my head around what's exactly needed

On the new server were going to add this which should deal with Canonicals, removal of upper case urls along with, removal of the forward slash

# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 33 - Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

would the above be best practice when doing a website name chnage

Thank you

[phranque]

what was the corresponding error message from the server error log for that 500 internal server error?

you should do all the canonicalization in one hop - doing the domain canonicalization in one step and further canonicalization in a subsequent step is not ideal.

RewriteCond %{HTTP_HOST} ^example.net$ [OR]
RewriteCond %{HTTP_HOST} ^www.example.net$

this would be better replaced by a single test:
RewriteCond %{HTTP_HOST} !^www.example.com$

the RewriteRule that follows should have a begin anchor on the pattern.

[kellyman]

Hi phranque

Thanks for the reply, if im honest i could not remember the actual error message

however we implemented the single line Below instead

RewriteEngine on
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

Im kind of at a bit of a loss here as i have to rely on my developer and he has always worked on windows servers so is not too familiar when it comes to the htaccess side, im learning and know a little

I personally thought that sending the site over to the new url and if any rogue non canonical's were sent they would be dealt with as and when on the new site

Mentioning the new site does that side look ok to you as i want to implement ASAP but worried that i cause more issues.

g1smd helped me out with that in another thread, but we wanted to test the old servers 301 before we implemented the htaccess and removed the PHP redirect as it was a lot more involved

[g1smd]

There should be no difference in a 301 redirect returned by PHP or by htaccess. A redirect is a redirect.

Things go wrong when a request results in an unwanted multiple step redriection chain. You should test a selection of canonical and non-canonical requests and look in detail at the Live HTTP Headers extension for Firefox for the results.

Redirecting old site page X to new site page Y should NOT pass through old site page Y or through new site page X on the way.

[kellyman]

There should be no difference in a 301 redirect returned by PHP or by htaccess. A redirect is a redirect.

Things go wrong when a request results in an unwanted multiple step redriection chain. You should test a selection of canonical and non-canonical requests and look in detail at the Live HTTP Headers extension for Firefox for the results.

Redirecting old site page X to new site page Y should NOT pass through old site page Y or through new site page X on the way.

Hi g1smd

Thats exactly what we did, the current 301 passes every single test, response headers are perfect too, everything looks textbook, maybe when i look mat the response codes i am missing something, but i get a 301 where they should be and a 200 for the correct page,

you mention in a reply you kindly gave me a few months back that when you helped me out on another question that when your requesting upper to lower case its one of the only occasions where the 301 is done after the rules as this would result in a multiple step redirection chain

However because of the way my guy has called the PHP redirects I am convinced they are chaining somewhere, as as his code is secret squirrel i never get to see if there is a mistake hence I am taking the approach of asking for a Htaccess to be implemented as 14 weeks for a simple name change is way too long, PR returned to inner pages home page nothing, i dont even show if you copy a line of text from the home page and put it into the search, it just seems the home page is very very weak

So deep down i know there is something not right, i never expect rankings back over night, but they are not even close to moving

So i was hoping from what i posted If i were to change the PHP to hataccess i have it exactly right then i can dismiss the fear i have that something in my web guys application is causing a redirect chain,

if what i opposed to do in the first post is best practice, as i really need to get this right now

Thanks again in adavnce

[lucy24]

Do you have access to the raw logs? When a visitor is human you will see the 301 immediately followed by a successful 200-- OR by another 301. Logs won't say what the request is being redirected to, but skip down to the next request from the same IP and you can see what they try next.

It's a little trickier with two domains because you're looking at two logs. But you can see patterns.

What you should see in the old site's logs is one 301 response per request, with no further activity on that site. In the new site's logs there should be an immediate 200 with no preceding 301.

[kellyman]

Hi Lucy

I don't have direct access to the logs, i can request them and have a look, and report back.

Going back to my original question though with this on the old server

RewriteEngine on
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

and this on the new server (thanks to g1smd) who supplied this for me

# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 33 - Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]


Is the correct way to fully redirect by 301 a site i have renamed.

Is there any errors that could possible cause a problem

BTW prior to me changing names my site was a PR4 DA of 62 and a PA of 68

currently home page is a shadow of those figures , and im not trying to replicate those however i don't think what i have at present is currently working as intended and being this my money site i am desperate to get some kind of normality back

really appreciate all the feedback and advice you guys have given.

[phranque]

if you request /index.php, for example, from the old server it will take two 301 responses before you get the Location: header containing the canonical url.

you can fix this in the .htaccess file on the old server without touching any php code because the request will be handled with a redirect by apache and the php script on the old server will never "know" it happened.

[lucy24]

with this on the old server

RewriteEngine on
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

and this on the new server (thanks to g1smd) who supplied this for me

# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 33 - Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

Nos. #31 and #32 need to be on BOTH sites so those requests get correctly redirected in a single step, no matter where the user came in. The same applies to ANY request involving any page in any form.
The only rule that won't be exactly parallel is your #33, the one that canonicalizes the host name. That one is only needed on the NEW server. The equivalent on the OLD server is a simple mopping-up that again comes after all the others:

RewriteRule (.*) http://www.example.com/$1 [R=301,L]

I've been using the words "site" and "server" interchangeably. But it really doesn't matter if it's the same server or a different one. The only thing that matters is the changed domain name. Even if both domains are sitting side by side in the same userspace on a shared server, redirects still have to go all the way outside and start over again as if they had never seen the server before.

[g1smd]

Good analysis in the previous posts.

Yes, most of the rules need to be on both servers.

However because of the way my guy has called the PHP redirects I am convinced they are chaining somewhere.

You should be internally rewriting to the PHP script so that the only redirect seen out on the web is the one issued by the PHP script.

If you inadvertantly redirect to the PHP script, there will be two redirects. This must be avoided.

In any case, it is easy to test. Request:
/
/index.php
/somepage
/somepage.php
/folder/somepage
/folder/somepage.php (and any other extensions)
on each of www and non-www on both the old and new site. You need a selection of the different types of URL on your site.

Look carefully at all of the results.

You should also test a variety of non-valid URLs. It is best that they all return 404 at the requested URL rather than redirect to a different URL and then return a 404, but sometimes that can't be avoided.

On a recent site migration I had a list of more than 800 test URLs to ensure that absolutely everything had been taken into account.

[kellyman]

Ok so pretty much there i think, just a little unsure if #30 comes last because if anywhere else it would result in a unwanted redirect

Really appreciate the input guys,

so on the old server we have this, i have put the redirect as the last call on the old server, hope i understood this correctly

RewriteEngine on


# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 30 - Redirect to new site
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

and new server we have this

# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 33 - Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

[kellyman]

Many thanks for those who helped me out in this

Implemented the first half on the old site and perfect,

Going to have to do the new site monday as we have to strip out all the old PHP 301's and document and rewrite them into htaccess

Or Is it possible to add the canonicals via htaccess but let a php script deal with some of the many individual pages or write these individual pages in htaccess as 1 file

I have seen examples of rewriting individual pages and looks easy enough to do but again many different scenarios .. ill check the threads later for examples

[phranque]

unless you need access to a database to determine the correct redirect, or it's an especially difficult or impossible pattern, you should probably do the redirect in the .htaccess file.

[g1smd]

In a recent site migration to extensionless URLs, requests for any of the old URLs with

?cat=<num>

and

?id=<num>

parameters were rewritten to a PHP script. The script looked up the new URL in an array and used the PHP

HEADER

directive to send a 301 redirect.

Those internal rewrites appeared right at the beginning of the htaccess file, a long way before the non-www to www redirect. To avoid a double redirect, the general canonical redirect further down the file also excluded requests that included these parameters from being redirected within the htaccess. This was done with a

RewriteCond

. Without this condition, the internal rewritten path would be exposed back out on to the web as a new URL when non-www URLs are requested.

The htaccess file also contained a variety of other redirects for old pages that have moved and various mis-spellings, etc. These appeared after the special rewrites and before the general canonical redirect.

The final chunk of code in the htaccess file contained the various internal rewrites that kick in when the new extensionless URLs are requested.

Some 55 000 URLs have been migrated. It took Google 3 months to completely get with the program. Traffic also trebled.

[kellyman]

Thanks again

I think ill stick with the htaccess as then i can see exactly what is being added

So which one of these is best i think from what i have read is the first one and if so can it at the top of the list so the internal redirects are shown first

RewriteRule ^oldpage$ http://www.example.com/test/newpage? [R=301,L]

Or

Redirect 301 /oldpage http://www.example.com/test/newpage?


I only have around 60 or so so not really that many

[g1smd]

Once you use a RewriteRule (from mod_rewrite) to do something, you must not use Redirect or RedirectMatch (from mod_alias) anywhere else in the same htaccess file. Rules are processed in "per module" order and NOT in the order listed in your htaccess file.

internal redirects

I don't understand what you mean.

There's external redirects, 301, 302 or 307, where the address bar URL changes and there are internal rewrites where the address bar URL does not change (which the Apache manual unhelpfully sometimes calls 'internal redirects').

[kellyman]

Hi when i mean Internal redirects i have pages were the names changed a

so for instance http://www.example.com/oldpage needs redirecting to http://www.example.com/newpage via 301 to preserve any links that were on those pages basically we renamed a category and added a new structure around a 7 months ago

[phranque]

internal/external in this terminology is relative to the requested url, not the hostname.

[lucy24]

Hi when i mean Internal redirects i have pages were the names changed a

so for instance http://www.example.com/oldpage needs redirecting to http://www.example.com/newpage

Just to confuse everyone, there are two sets of terminology. Apache calls a rewrite an internal redirect ... and who's going to tell 'em not to? ;) In this forum we try to use double markedness* so it's either "internal rewrite" (for things that happen behind the scenes without the user's knowledge) or "external redirect" (for things that result in a visible change in the address bar).

Here's an important thing to internalize. It falls under "Oh, oops, I forgot that you might not have known this already". Once you've got a Redirect, whether 301 or 302 or
unmarked [R], it makes no difference whether the domain name is the same or different. The browser can't just go down the hall and knock on a different door. It has to go all the way outside, get past the doorman again, show its ID to the security guard again, explain itself to the elevator operator again, et cetera. And none of these people will say "Oh, yeah, I remember, you were here two nanoseconds ago."

Unless the owner of the building has put in place a php script that says "If someone tries to engage in such-and-such action, slap on a brand-new set of rules for any subsequent visits from anyone matching his description." But we're not there yet.


* aka "redundancy", but the linguistic term is more useful here.

[kellyman]

OK

Kind of confused with the terminology but kind of have the idea,

so if i wanted the second part of this htaccess added to the new site which i will do Monday, is this correct allowing for already redirected pages


RewriteEngine on

# 2 Redirect Individual renamed pages

RewriteRule ^oldpage1$ http://www.example.com/test/newpage1? [R=301,L] 
RewriteRule ^oldpage2$ http://www.example.com/test/newpage2? [R=301,L] 
RewriteRule ^oldpage3$ http://www.example.com/test/newpage3? [R=301,L] 
RewriteRule ^oldpage4$ http://www.example.com/test/newpage4? [R=301,L] 
RewriteRule ^oldpage5$ http://www.example.com/test/newpage5? [R=301,L] 
RewriteRule ^oldpage6$ http://www.example.com/test/newpage6? [R=301,L] 
RewriteRule ^oldpage7$ http://www.example.com/test/newpage7? [R=301,L] 
RewriteRule ^oldpage8$ http://www.example.com/test/newpage8? [R=301,L] 
RewriteRule ^oldpage9$ http://www.example.com/test/newpage9? [R=301,L] 
RewriteRule ^oldpage10$ http://www.example.com/test/newpage10? [R=301,L] 
RewriteRule ^oldpage11$ http://www.example.com/test/newpage11? [R=301,L]

# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 33 - Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

[g1smd]

Looks OK.

If any of the old page names contain a period, don't forget to escape it.

[kellyman]

Thanks for all the replys and the time taken to get me this far it really is appreciated.

I do have another question in regard to htaccess but ill write another post next week after a bit of research..

[kellyman]

Hi Guys

Implemented the code and all looks fine, however since implemented, i have a few functions within the site that call on a php function i.e a login, that are not now working

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

How Could i modify the above code to remove the .php (just worried about funcionality by just removing the .php

[g1smd]

You'll either need to exclude requests for login.php from being redirected, or else amend your form submit to POST to the extensionless URL, and implement a rewrite to connect the extensionless request to the internal .php file.

[kellyman]

I Think i may have explained that incorrectly , we can actually access all pages but when any forms on the site are posted, they are posted to form-process.php file which the htaccess is preventing the processing of any forms.

Or can it just be a line added

RewriteRule ^/form-process\.php$ - [L]

[g1smd]

You explained it perfectly adequately. This should do it, on all the rules that interfere with form submission:

RewriteCond %{REQUEST_URI} !^/form-process\.php

or even

RewriteCond %{THE_REQUEST} !^POST

[kellyman]

You explained it perfectly adequately. This should do it, on all the rules that interfere with form submission:

g1smd

Your a star , many thanks for all your help and your quick response.

Ill get this line added first thing tomorrow,

Hopefully i can report back in a few weeks and have some positive news in regard to the site and its rankings

Thanks to the others who helped too it really is appreciated.

[lucy24]

or even

RewriteCond %{THE_REQUEST} !^POST

Can you use %{REQUEST_METHOD} or is that one of those things that potentially changes in midstream?

[g1smd]

Yeah, I suppose you can; but I don't think I ever have. :)

[kellyman]

Hi

Added that and it did not work, does it need to go into a certain position in the ordering

the actual command is run.php we have to let execute

RewriteEngine on

# 2 Redirect Individual renamed pages

RewriteRule ^oldpage1$ http://www.example.com/test/newpage1? [R=301,L]
RewriteRule ^oldpage2$ http://www.example.com/test/newpage2? [R=301,L]
RewriteRule ^oldpage3$ http://www.example.com/test/newpage3? [R=301,L]
RewriteRule ^oldpage4$ http://www.example.com/test/newpage4? [R=301,L]
RewriteRule ^oldpage5$ http://www.example.com/test/newpage5? [R=301,L]
RewriteRule ^oldpage6$ http://www.example.com/test/newpage6? [R=301,L]
RewriteRule ^oldpage7$ http://www.example.com/test/newpage7? [R=301,L]
RewriteRule ^oldpage8$ http://www.example.com/test/newpage8? [R=301,L]
RewriteRule ^oldpage9$ http://www.example.com/test/newpage9? [R=301,L]
RewriteRule ^oldpage10$ http://www.example.com/test/newpage10? [R=301,L]
RewriteRule ^oldpage11$ http://www.example.com/test/newpage11? [R=301,L]

# 10 - ignore Page
RewriteCond %{REQUEST_URI} !^/run\.php


# 31 - Redirect index requests on new server
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index(\.(php|html?))?
RewriteRule ^(([^/]+/)*)index(\.(php|html?))?$ http://www.example.com/$1? [R=301,L]

# 32 - Redirect .php requests to extensionless URL
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(([^/]+/)*[^/.]+)\.php
RewriteRule ^(([^/]+/)*[^/.]+)\.php$ http://www.example.com/$1? [R=301,L]

# 33 - Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]