Does Rewrite -> PHP header chaining return a real 301 ?

Redirect chain, what status code is actually returned to the search engine?

[GodfatherX]

Hi there,

Problem:
I need to re-route old Joomla 1.0 URLs to the new URL created by Joomla 1.5. The problem is, that I have 10,000 articles and there is no way of maintaining a 1:1 redirect list nor a rewrite by regular expression, since Joomla 1.5's SEF URLs are based on the article title and other factors, which are not provided by the old URL.

Example:
OLD: http://www.example.com/content/view/34554/123/
NEW: http://www.example.com/reviews/entertainment/34554-the new-bose-speakers

Only a certain Joomla library can determine via db-requests what the new URL will be. So I thought that I would redirect the incoming OLD URL to a php script (new_sef_route.php), that finds out what the NEW SEF URL is and then redirects a second time to that final URL.

But the big question for me is:
What status code is actually returned to the search engine, where the link was originally clicked?

So far, I have the following setup:

// In .htaccess
----------------------------------

RewriteRule content/view/([0-9]{1,5})/([0-9]{1,3}) http://www.example.com/new_sef_route.php?id=$2 [R=301,L] 
RewriteRule content/view/([0-9]{1,5}) http://www.example.com/new_sef_route.php?id=$1 [R=301,L] 

// In new_sef_route.php
----------------------------------

<?php 
$id = (int) $_GET["id"]; 
  
...some computations to get the new URL and store it in $newurl... 
  
/*** Redirect permanent 301 to the destination URL: ***/ 
header ('HTTP/1.1 301 Moved Permanently'); 
header("location: http://www.example.com".$newurl); 
exit(); 
?> 

Does anyone know what status code is returned and whether this setup would be the right way to do it.
And also are there any search engine problems that are obvious and that I have overlooked?
Obviously I intend to tell the search engines to replace the OLD URL with the NEW URL in their index; the whole purpose of this exercise.

Would be very thankful for your help.

Markus

PS:
I apologize for the unintended links. Somehow this forum post automatically links the [blablah...] lines, I couldn't find where to turn this off.

[edited by: GodfatherX at 11:10 pm (utc) on Oct. 26, 2009]

[g1smd]

Don't redirect to a script at a new URL, instead use an internal rewrite to that script.

The script should then send a 301 redirect pointing to the new URL; the new URL should be FQDN.

That will return a single 301 redirect from old to new URL.

Your original scheme would return a double redirect. You should avoid that.

[GodfatherX]

Thanks g1smd, for this blazingly fast reply. :-)

So, I should simply change my htaccess line to this (leaving out the R=301)?

// In .htaccess
----------------------------------

RewriteRule content/view/([0-9]{1,5})/([0-9]{1,3}) http://www.example.com/new_sef_route.php?id=$2 [L] 
RewriteRule content/view/([0-9]{1,5}) http://www.example.com/new_sef_route.php?id=$1 [L] 

I guess the php script can stay as it is, since it must issue the 301.

Markus

PS: I already edited my post to example dot com, but you were so incredibly fast with your response. :-)

[jdMorgan]

That's now a chained 302->301 redirect. Leave out the http+domain name, so that you internally rewrite requested URLs to a script filepath. If possible, fully-anchor your URL-path pattern to avoid unexpected operation, and if there is a slash on the end of the URLs, include that in the pattern:

RewriteRule ^content/view/([0-9]{1,5})/([0-9]{1,3})/$ /new_sef_route.php?id=$2 [L]
RewriteRule ^content/view/([0-9]{1,5})/$ /new_sef_route.php?id=$1 [L] 

Having done that, completely flush (delete) your browser cache, and test your rewriterule+redirection script using the Live HTTP Headers add-on for Firefox (or similar) to be sure that you get a single 301-Moved Permanently redirect from the old URL to the new.

Jim

[GodfatherX]

Thanks Jim.

That was very helpful.
The old URLs can or cannot have a trailing slash.

I will look for the headers add-on, too.

Markus

[g1smd]

The ? qualifier will allow the slash to be optional.

You can have optional components when the request is going to be redirected, as the user will end up at a single canonical URL from wherever they started.

You absolutely cannot have optional component parts for the URL that directly delivers the content. That's a Duplicate Content issue.

[jdMorgan]

If both types of URLs may have an optional trailing slash, then end the patterns with "/?" instead of just "/" as shown above. And note that this is recommended *only* if all such URLs will be subsequently redirected (in this case, by your script); Otherwise, it creates a potential duplicate-content problem.

Jim

[added] Oops! Where was g1smd's reply when I last checked this thread? Ya gotta be fast to keep up with him... :o [/added]

[edited by: jdMorgan at 12:56 am (utc) on Oct. 27, 2009]

[g1smd]

LOL. But at least we both gave the same answer. :)

The forum is all yours; it's nearly 1 a.m. so I am off to bed.

[GodfatherX]

Thanks guys,
but I am a bit confused with the double content issue.

OLD: http://www.example.com/content/view/34554/123/
OLD: http://www.example.com/content/view/34554/123
OLD: http://www.example.com/content/view/34554/
OLD: http://www.example.com/content/view/34554
OLD: http://www.example.com/index.php?option=com_content&task=view&id=34554&Itemid=123
OLD: http://www.example.com/index.php?option=com_content&task=view&id=34554

Any of the URLs above would display the same content in Joomla 1.0. This is not something I am doing, but just how Joomla works. And I have no control over other people's websites and how they are linking to us.

On the new installation (Joomla 1.5), I plan to redirect all these URLs to this _ONE_ URL returning a 301.

NEW: http://www.example.com/reviews/entertainment/34554-the new-bose-speakers

Is that considered double content?
As I understand, with 301 I am trying to tell the search engine that this is all the same page so it can replace all these possible multiple links in its index with the ONE final URL.
As I look at it, I am trying to do the search engine a favor.
Is that going to be penalized?

Markus

[jdMorgan]

You will be penalized only if you are indeed 'cheating' and deserve a penalty. Other than that, there are practically *no* penalties, and I'd like to bury that term. You may trigger a filter, but even doing that is rare and almost always occurs only when 'you get caught' doing something wrong... There is far too much fear and unreasonable doubt among honest Webmasters, but Google can't really fully-describe their true 'penalties' without giving away too much information to the real cheaters...

All I can say is th in order to get a real penalty, you have to be doing something really wrong or make *a lot* of serious mistakes; Google wants to index *all* of the useful information on the Web, even the information from badly-implemented sites. So they try hard not to penalize unless it's really deserved.

Just as an example, most Web site's "home page" can be reached using 16 different URLs, and most of them do OK despite that. They might do a little better by canonicalizing www versus non-www, redirecting FQDN to non-FQDN, removing port numbers if present, and canonicalizing "index.html" or "default.php" to "/", but they don't suffer too much if at all; Most of their Webmasters never even notice anything wrong...

Oh, and if you add in https versus non-https, then the number of "home pages" doubles again to 32... Add query string varitions, and the number could well be practically infinite.

Using a 301 redirect to 'correct' old or bogus URLs is not doing anything wrong *unless* you continue to publish links to those old and bogus URLs -- and even then, a reasonable number of errors will very likely be forgiven... real penalties are *really* for real cheaters.

The most common problem that might look like a penalty is that by publishing multiple URLs that point to the same content, you set up each of those URLs to compete with the others for inbound links and ranking, just as if each of them were competing with similar-content URLs on completely-different Web sites. So of course, none of them will do as well as any one would do if it were the only URL leading to that same content...

So forget 'penalties,' but don't compete with yourself... 301-redirect all of those variant URLs to one URL, and you'll get the best results.

Note that in order to redirect the URLs that look like script filepaths, you will likely need to check the original client request before redirecting. Only in the case where the client is requesting that filepath/URL-path should you redirect. You *do not want* to redirect if the client requested a different URL that was then rewritten to that filepath!

Jim

[GodfatherX]

Thanks Jim and g1smd, for all you help.

You guys are awesome.

[GodfatherX]

Hello guys,

this is a follow-up question:
I implemented everything as Jim and g1smd suggested and it works - or does it?
For the eyes of the visitors, everything works as intended, but I am worried about the search engines.
I would appreciate if you could spare a minute to look over the following header outputs. Thanks.

(I hope it's not considered inappropriate to post this much verbose data, but I thought it helps to identify the problem. Large fonts are only used for better readability)

MY IMPLEMENTATION:
-------------------------------------------

----- in .htaccess ------ 
RewriteRule ^content/view/([0-9]{1,5})/([0-9]{1,3})/new_sef_route.php?id=$1 [L] 
RewriteRule ^content/view/([0-9]{1,5}) /new_sef_route.php?id=$1 [L] 
  
  
----- in the redirecting script /new_sef_route.php ------ 
if (!$found_new_url) 
{ 
 header ('HTTP/1.1 404 Page Not Found'); 
 header('Location: ['.$_SERVER["HTTP_HOST"].'...]  
 exit(); 
}  
else 
{ 
 header ('HTTP/1.1 301 Moved Permanently'); 
 header("location: [".$_SERVER["HTTP_HOST"].$found_new_url);...]  
 exit(); 
} 

HEADER OUTPUT:
----------------------------------------
The Firefox "Live HTTP Headers" Addon shows these confusing results:

a) In case of a valid old URL redirecting to a valid new URL, this is the first status response:
================ Begin of Headers ======================================
http://www.example.com/content/view/44455/140/

GET /content/view/44455/140/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.8,de-de;q=0.5,de;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: ITXT_TTV=1; __qca=1201721796-72528666-56435590; ebNewBandWidth_.www.example.com=501%3A1253216562499;

HTTP/1.x 301 Moved Permanently
Date: Mon, 02 Nov 2009 21:57:14 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.2.9
X-Powered-By: PHP/5.2.9
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Location: http://www.example.com/electronics/44455-sony-strikes-again
Content-Length: 0
Connection: close
Content-Type: text/html
----------------------------------------------------------
http://www.example.com/electronics/44455-sony-strikes-again

GET /electronics/44455-sony-strikes-again HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.8,de-de;q=0.5,de;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: ITXT_TTV=1; __qca=1201721796-72528666-56435590; ebNewBandWidth_.www.example.com=501%3A1253216562499;

HTTP/1.x 200 OK
Date: Mon, 02 Nov 2009 21:57:14 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.2.9
X-Powered-By: PHP/5.2.9
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Mon, 02 Nov 2009 21:57:15 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
----------------------------------------------------------
...continues loading a lot of ad stuff from here on...
================ END of Headers (excerpt) ======================================

b) In case of an invalid old URL redirecting to custom 404 page, this is the first status response:
================ Begin of Headers ======================================
http://www.example.com/content/view/77777/140/

GET /content/view/77777/140/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.8,de-de;q=0.5,de;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: ITXT_TTV=1; __qca=1201721796-72528666-56435590; ebNewBandWidth_.www.example.com=501%3A1253216562499;

HTTP/1.x 302 Found
Date: Mon, 02 Nov 2009 22:03:34 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.2.9
X-Powered-By: PHP/5.2.9
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Location: http://www.example.com/index.php?option=com_content&view=article&id=3
Content-Length: 0
Connection: close
Content-Type: text/html
----------------------------------------------------------
http://www.example.com/index.php?option=com_content&view=article&id=3

GET /index.php?option=com_content&view=article&id=3 HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.8,de-de;q=0.5,de;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: ITXT_TTV=1; __qca=1201721796-72528666-56435590; ebNewBandWidth_.www.example.com=501%3A1253216562499;

HTTP/1.x 200 OK
Date: Mon, 02 Nov 2009 22:03:34 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.2.9
X-Powered-By: PHP/5.2.9
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Mon, 02 Nov 2009 22:03:35 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
----------------------------------------------------------
...continues loading a lot of ad stuff from here on...
================ END of Headers (excerpt) ======================================

c) In case of a valid old URL (clicked on from a Google result) redirecting to a valid new URL, this is the first status response:
================ Begin of Headers ======================================
http://www.example.com/content/view/44455/140/

GET /content/view/44455/140/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.8,de-de;q=0.5,de;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: [google.com...]
Cookie: ITXT_TTV=1; __qca=1201721796-72528666-56435590; ebNewBandWidth_.www.example.com=501%3A1253216562499;

HTTP/1.x 301 Moved Permanently
Date: Mon, 02 Nov 2009 21:59:21 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.2.9
X-Powered-By: PHP/5.2.9
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Location: http://www.example.com/electronics/44455-sony-strikes-again
Content-Length: 0
Connection: close
Content-Type: text/html
----------------------------------------------------------
http://www.example.com/electronics/44455-sony-strikes-again

GET /electronics/44455-sony-strikes-again HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.8,de-de;q=0.5,de;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: [google.com...]
Cookie: ITXT_TTV=1; __qca=1201721796-72528666-56435590; ebNewBandWidth_.www.example.com=501%3A1253216562499;

HTTP/1.x 200 OK
Date: Mon, 02 Nov 2009 21:59:21 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.2.9
X-Powered-By: PHP/5.2.9
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Mon, 02 Nov 2009 21:59:22 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
----------------------------------------------------------
...continues loading a lot of ad stuff from here on...
================ END of Headers (excerpt) ======================================

Google result listing:
-------------------------------------------
d) Also here is the output of the Google search result listing, showing that they did not replace the old URL with the new URL in their index, but that they list both URLs now - the new URL as an indented subset of the old URL:
(is this the much talked about double content issue?)
================ Begin of Google Result (relevant excerpt) ======================================

Example Site - Sony strikes again 
Oct 28, 2009 ... Rich Audible, Sony has once again done it and introduced a great pair of speakers 
that outperform even the best models of ... 
www.example.com/content/view/44455/140/ - Cached - Similar - 
  
 RightNow's pre-emptive strike 
 Rich Audible, Sony has once again done it and introduced a great pair of speakers that outperform 
 even the best models of manufacturers like ... 
 www.example.com/electr.../44455-sony-strikes-again - Similar - 
  
 + Show more results from www.example.com 
  

================ END of Google Result (relevant excerpt) ======================================

THE QUESTION:
-------------------------------------------
Is the the implementation above correct?
This might be more appropriate in the php forum, but I wnated you guys to examine the headers.
a) It seems that in a) a proper 301 is sent, but as you can see in d) Google didn't replace the data, only list it as a subset.
b) The redirect of the header doesn't send a 404, but a 302 instead, as far as i can see. What's wrong?
c) Same as a). Although there is one more redirect done by Google, there is a 301, but it doesn't seem to take effect.

What do I do wrong?

I appreciate any help.

Markus

[edited by: jdMorgan at 1:38 am (utc) on Nov. 3, 2009]
[edit reason] Snipped Google redirect & truncated cookie to shorten & prevent side-scrroll. [/edit]

[TheMadScientist]

This line is the issue:
(With b anyway.)

if (!$found_new_url)
{
header ('HTTP/1.1 404 Page Not Found');
header('Location: http://'.$_SERVER["HTTP_HOST"].'/index.php?option=com_content&view=article&id=3');
exit();
}

A 302 is an 'undefined' Redirect, which yours is... You are sending people to a new location without defining the Redirect as permanent, or anything else. It's basically the same issue as discussed earlier with Rewriting to a Redirect script, rather than Redirecting to it.

The solution to serving a proper 404 is: rather than externally redirecting to the 404 page, include it...

if (!$found_new_url)
{
header ('HTTP/1.1 404 Page Not Found');
$option='com_content';
$view='article';
$id='3';
include $_SERVER['DOCUMENT_ROOT'] . 'index.php';
exit();
}

[jdMorgan]

And as for Google not updating your old URLs to new, I'd say, take a two-month vacation, and then check again... Don't expect this to happen instantly, don't be surprised if you see it "switch back and forth" for awhile, but literally, ignore this for 30 to 180 days... If it's still wrong next March, then worry. In the meantime, the fix posted above is all you need to 'prove' that it will work, given your posted headers log.

Jim

[g1smd]

Just to add, for a URL to be returning a true 404 error, the originally requested URL should directly return the 404 header.

That is, if the requested URL redirects to some other place then the requested URL is NOT returning a 404 - redirect codes are 30x.

It is not sufficient for the requested URL to redirect so some other place and to then rely on that other place returning a 404 header.

Additionally, any sort of 302 redirect can be very dangerous to site health. Avoid those at all times unless there is a very good reason to use them (almost never).

[GodfatherX]

Thanks Guys,

TheMadScientist,
I can't just "include" the variables and load the index.php, as it relies on the variables being available in the $_REQUEST array. The CMS doesn't work this way.

Jim,
Thanks, I guess I have to be patient and find out only after months whether I made a fatal mistake or not. ;-)

g1smd,
Thanks, that is very helpful, but I thought according to Jim and your instructions above, that I don't perform a redirect but a rewrite in the first step (.htaccess). Therefore my "404 redirect" attempt in script (see IMPLEMENTATION above) is the first actual redirect.
But it still doesn't seem to be returned.

[jdMorgan]

Nope, your server headers checks will tell you immediately whether you've made a mistake or not. If you see a 30x response when a 404 is required, that's a big problem. If you see multiple consecutive redirects instead of a single redirect to the correct URL, that's another. If a redirect exposes an internal script filepath in the URL, that's yet another. Evaluating these headers, you're looking at exactly what the search engine robots will see when they examine your server responses.

The only 'waiting' will be to determine when the search engines have picked up your thoroughly-tested and known-good corrections.

Here the 404 handling seems most problematic. However, you absolutely must NOT redirect to a 404 page -- or to any other error page which should in fact return an error response code. If you do, you're likely going to sink your site. If necessary, forgo using any scripting to generate 404 responses or use a separate (simple) script to do it, by-passing the complex and restrictive CMS code completely.

Redirecting when an error response is required creates some very nasty problems, some of which are exploitable by your competitors, so this is not to be trifled with no matter how 'inconvenient' the proper solution may be; The end results of such exploitable errors can and do put sites out of business.

Jim