301 using .htaccess including subdomains

[Seawitch]

Hi all,

I'm puzzled, as the techs at the hosting company made the changes, but they're not working. Between them and me, we've all been beating our heads in to a wall. LOL.

In any case, what I'm looking to do is to redirect 301 from the old domain to the new, INCLUDING the subdomains (the structure is identical, domain to domain), while having both hosted on the same dedicated server.

Here is what I have:

Options +FollowSymlinks

DirectoryIndex index.html index.htm index.php

RewriteEngine on
RewriteBase /
#RewriteRule !redirect.html /redirect.html
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_HOST} ^(.*)example\.net [NC]
RewriteRule (.*) http://%1example.org%{REQUEST_URI} [R=301,L]

I'm sure there's an error somewhere, but where?

[JD_Toims]

The caching really has to be by your server itself or a [misbehaving] down-stream proxy server between the requesting location and your server -- If it was your browser it would be obvious. The WebmasterWorld HTTP Response Checker would *definitely* receive the 301 redirect status via PHP even if it doesn't send a host header and your browser didn't get a redirect, because the tool keeps no cache, meaning every requests it makes is "fresh" -- That's why I wanted to see what the tool came back with and what happened when we put a redirect on the PHP page itself.

Hopefully someone else has run into something similar before and has a solution, because the response code you're getting [200 OK] with the PHP code on the page when you're making a request with a cache that has to be empty [WebmasterWorld HTTP Response Checker] isn't something I've run into before -- There should be a redirect served either via .htaccess/mod_rewrite or via PHP if the .htaccess/mod_rewrite doesn't "kick-in" for some reason.

[lucy24]

or a [misbehaving] down-stream proxy server between the requesting location and your server

Oh, YUK. That is a real possibility and may be an insoluble problem. Some ISPs are notorious for remote-caching, and no power on earth will get them to disgorge a fresh copy of a page if they don't feel like it. You think you're refreshing, but all you're getting is whatever the ISP has in its cache. Locally it's one reason I'm staying with DSL instead of going back the the cable company. Naming no names.

Is all this happening on your live site? If so, you may not be able to stop caching. On my test site I added a nifty set of lines that say

ExpiresActive On [this line may not be necessary depending on server settings]
ExpiresByType text/html "access"
ExpiresByType text/php "access"

so I never have to empty the browser cache.

This excerpt from the previous page

RewriteEngine on
#RewriteRule !redirect.html /redirect.html
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_HOST} ^(.*\.)?old\.net [NC]
RewriteRule ^(.*)$ http:/ /%1new/$1 [R=301,L]
RewriteCond %{HTTP_REFERER} !^http://www2.old.net/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www2.old.net$ [NC]

made me uneasy because it seems to stop in mid-sentence. What's the rule that the last two conditions belong to? And, for that matter, why do you need /.*$ and $ on separate lines? They're the same thing-- and can be expressed with no closing anchor at all.

[JD_Toims]

Thanks for chiming in Lucy24 -- The ruleset you quoted has been replaced with the following:

RewriteEngine on

RewriteCond %{HTTP_HOST} ^([^.]+\.)?example\.net$
RewriteRule .? http://%1example.org%{REQUEST_URI} [R=301,L]

###

And the PHP from a previous post is also on the PHP page:

###

<?php
error_reporting(0);
header('Location: http://subdomain.new-domain.org'.$_SERVER['REQUEST_URI'].'?'.$_SERVER['QUERY_STRING'],TRUE,301);
exit;

// Everything else below here.

###

It's why I'm so confused as to why there's not a redirect being received even by the HTTP Header Checker here -- I think you might be on to something along the same lines as an ISP cache here:

Oh, YUK. That is a real possibility and may be an insoluble problem. Some ISPs are notorious for remote-caching, and no power on earth will get them to disgorge a fresh copy of a page if they don't feel like it. You think you're refreshing, but all you're getting is whatever the ISP has in its cache.

I'm not sure it's the ISP though, because the HTTP Header Checker here is receiving the same 200 OK from the requested location and should be ISP independent, so I'm leaning to either the host server itself caching or a "close to the host" down-stream proxy server caching even when all requests should be revalidated and served fresh with the Expires being set to a past date and the Cache-Control being set to no-store, no-cache, must-revalidate, etc.

The expires *should* take precedence over the Cache-Control in a properly functioning server/proxy-server, so having it set to a past date and getting the same response code via browser and the HTTP Header Checker here makes me think there's something wrong at [or close to] the host -- Any thoughts or experience with this type of thing?

[edited by: JD_Toims at 2:37 am (utc) on Sep 15, 2013]

[Seawitch]

It's quite possible it's only on my end as far as the wonky caching. I had a couple other people I know check the links for redirects working or not working, and the subdomain redirects are fine, just not the variable links. Whereas when I check, some of the subdomains redirect, and some do not.

It may very well be a combination then. I know there are crons set up to run on my server nightly, one of which resets everything and empties any caches. So, I'll see what it does tomorrow.

[JD_Toims]

...one of which resets everything and empties any caches. So, I'll see what it does tomorrow.

Ah, it could be something to do with your server setup then -- Please let us know what results you receive tomorrow when you check, because there's definitely something "odd" going on here.

[Seawitch]

If that doesn't resolve it, I can always open a ticket and have one of the techs check and see if there's a cache file that needs emptied, or if the server requires a restart.

[Seawitch]

So far, what's redirecting are the roots of the domain and subdomains, not the variable file links. However, I'm waiting for someone else to come online and check for me, as I'm not getting the same results as everyone else.

This still with the header info

HTTP/1.1 200 OK
Date: Sun, 15 Sep 2013 18:16:34 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=7rt4qusplbe327pt43gs1pill6; path=/
Vary: Host,Accept-Encoding,User-Agent
Connection: close
Content-Type: text/html

So I will definitely be opening a ticket with my host and giving them that information.

[aakk9999]

To see if it is server/ISP caching, you could perhaps do the following:

1) On the old domain, copy story.php (with JD_Toims php code in it) as story-test1.php

2) Request header info for http://subdomain.old-domain.net/story-test1.php?var=val

That page, as it is new and was never requested before, won't be cached (if there is ISP caching). So you *should* get 301 Response.

If the above returns 301, then we need to find out where 301 came from, php or .htaccess. Now do the following:

1) On the old domain, copy story.php (without the JD_Toims' php code in it) as story-test2.php

2) Request header info for http://subdomain.old-domain.net/story-test2.php?var=val

If the response code is 301, then .htaccess redirect works and the problem is ISP caching. If the response code is 200 OK, then there is some conflict with .htaccess on top of having a caching issue.

[Seawitch]

Test 1

HTTP/1.1 404 Not Found
Date: Sun, 15 Sep 2013 23:30:12 GMT
Server: Apache
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

Test 2

HTTP/1.1 404 Not Found
Date: Sun, 15 Sep 2013 23:31:20 GMT
Server: Apache
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

Original file and link

HTTP/1.1 302 Moved Temporarily
Date: Sun, 15 Sep 2013 23:31:53 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=m56eghbi1j6mrq94v8m323ut84; path=/
Vary: Host,Accept-Encoding,User-Agent
Location: http://sub.new.org/cookiepage.php
Connection: close
Content-Type: text/html

[aakk9999]

Can you do the same test, but make a copy of the story.php as story-test3.php (with JD_Toims php code) and story-test4.php (without JD_Toims php code) in the NEW domain folder? And then request the OLD domain URLs when checking headers.

Request header info for:
http://subdomain.old-domain.net/story-test3.php?var=val
http://subdomain.old-domain.net/story-test4.php?var=val

I am wondering whether the OLD domain is resolving to NEW domain folder because of some setup. It seems it does not resolve to the OLD domain folder, otherwise you would have 301 from the first test and either 200 or 301 from the second.

<added>Original file and link - are you saying that "story.php" is now redirecting? Or something else?

BTW, it has 302 redirect, it should be 301
</added>

[phranque]

use nslookup to verify that the various hostnames are resolving to the correct IP address(es).

[Seawitch]

Rechecked, the sites are resolving properly

link using the file, no test paramter

HTTP/1.1 302 Moved Temporarily
Date: Mon, 16 Sep 2013 01:24:39 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=efrodb0ugtoo0djseqpdrru630; path=/
Vary: Host,Accept-Encoding,User-Agent
Location: http://sub.newdomain.org/form_cookie.php
Connection: close
Content-Type: text/html

test 3

HTTP/1.1 301 Moved Permanently
Date: Mon, 16 Sep 2013 01:29:58 GMT
Server: Apache
Vary: Host,Accept-Encoding,User-Agent
Location: http://sub.newdomain.org/story-test3.php?no=600054278?no=600054278
Connection: close
Content-Type: text/html

test 4

HTTP/1.1 302 Moved Temporarily
Date: Mon, 16 Sep 2013 01:30:55 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=sh8ftsk91h14kk6tto0dfhbeh5; path=/
Vary: Host,Accept-Encoding,User-Agent
Location: http://sub.newdomain.org/form_cookie.php
Connection: close
Content-Type: text/html

this is as instructed, with the test files added to the new domain

[JD_Toims]

There's an issue with your mod_rewrite in the *new* domain .htaccess files if you're getting a 302 redirect -- The /story-test3.php page is the only one issuing the correct header [meaning it's likely the PHP issuing the redirect]. The /story-test4.php file is being redirected but as a 302 rather than a 301, which means "temporary" or "undefined", so it's better to go with a 301.

Note About the Preceding: Since the "302 redirect bug" Google has broken protocol (the only "blatant, purposeful" breach of HTTP protocol I know of from Google -- privacy breaches are another story for another thread lol) and is now treating 302 redirects almost exactly the same as 301s. There may be more time before a 302 is treated as permanent when compared to a 301, but eventually, if a 302 is in place long enough to meet their threshold it will be treated as a 301.

[edited by: JD_Toims at 3:32 am (utc) on Sep 16, 2013]

[Seawitch]

Okay, what I'm going to do then, is link this topic to my hosting company tech, so he can get to the root of it. Once that's addressed, the rest should work correctly?

[JD_Toims]

Okay, what I'm going to do then, is link this topic to my hosting company tech, so he can get to the root of it. Once that's addressed, the rest should work correctly?

Yes and sounds like a good plan to me.

[aakk9999]

Emphasis mine, from your post way up in this thread:

In .htaccess, at the root of the old domain.

Since the request for these test files you placed on old domain respond with 404, and the request for story-test3.php returned the correct 301 (as JD_Toims said, almost certainly issued by PHP), I think these lines in .htaccess must be in the .htaccess of the subdomain folder on the NEW domain.

Because it seems this is where requests are resolving to.

What is in .htaccess that is in subfolder of the NEW domain (if there is one)?

Perhaps you could put these lines JD_Toims has given you in the .htaccess that is in the new subdomain and try again checking headers for story-test4 and for your existing file?

[Seawitch]

So......I should put the 301 redirects for the old domain in the new domain's .htaccess? Won't that make things loop?

[aakk9999]

No it won't because of the RewriteCond (condition for redirect):

RewriteCond %{HTTP_HOST} ^([^.]+\.)?example\.net$
RewriteRule .? http://%1example.org%{REQUEST_URI} [R=301,L]

Which says: Only do the following line of redirect IF the request comes from the OLD domain, then redirect to new domain.

In the above example, any request for the example.org (new domain) will skip the above redirect. Only requests for example.net (old domain) will execute redirects.

[aakk9999]

And another thing
For this test, put these lines towards the top of .htaccess of the NEW subdomain, after the line:

Rewrite Engine On

This is make sure the rule gets executed (just in case that something else in the .htaccess of the NEW subdomain does not come into the play first, so the rule may not get executed).

If we then see the redirect working, then we can check your .htaccess of the NEW subdomain to find the best place to put them.

[Seawitch]

still getting the same results. I'm thinking there HAS to be something server side I don't have access to that needs changed.

[aakk9999]

Have you put these lines at the top of the .htaccess of the new subdomain? So that the .htaccess in the NEW subdomain has it as follows:

RewriteEngine on

RewriteCond %{HTTP_HOST} ^([^.]+\.)?example\.net$
RewriteRule .? http://%1example.org%{REQUEST_URI} [R=301,L]

of course, replacing example.net and example.org with your old and new domain names.

[aakk9999]

I am also looking your Test 4 and "link using the file, no test paramter " from few posts above, where you are getting 302 redirect.

What is interesting is the Location:

Location: http://sub.newdomain.org/form_cookie.php

If you have a request for story.php and story-test4.php, I would expect the location to be

For link using file, no test parameter
Location: http://sub.newdomain.org/story.php

For Test4
Location: http://sub.newdomain.org/story-test4.php

I presume you only exemplified domain name in your results, and that /form_cookie.php is the path after the domain name that you got in your header check response

Since both requests are doing 302 redirect to the same "Location", it seems there is some setting in place to send requests for the old subdomain to /form_cookie.php script

This could be on your server config or in .htaccess of the new subdomain.

[Seawitch]

yep, and it still is not changing the url to redirect old to new

[lucy24]

put the 301 redirects for the old domain in the new domain's .htaccess?

Do requests for one domain pass through the other domain's htaccess?

Each domain has DNS information that points requests to the right place. But requests don't levitate straight to their destination. They have to come in the front door (the server) and then move through one or more layers of physical directories. Along the way they have to obey the directives of the config file as well as any htaccess files they happen to meet.

If analogies help: When you go to someone's office you're not allowed to climb in their back window, even if it opens conveniently onto the fire escape. You have to get past the doorman and the elevator attendant and the receptionist and the personal assistant and...

[Seawitch]

the config file that would I assume control how .htaccess behaves is something I don't have access to.

[aakk9999]

I think now to do what you suggested and JD_Toims also said - point your host support to this thread. There is enough info and test results in here that they should figure out what is wrong.

[Seawitch]

So how we got it to work, finally, was by adding an .htaccess file to each subdomain. Apparently that's how the server is set up or something, as that's what the hosting company had me do.

What's weird is that I had to add the .htaccess file to the NEW domain's subdomains, not the old, in order for this to work.

Thanks again for all the help, you guys have been awesome!

[aakk9999]

What's weird is that I had to add the .htaccess file to the NEW domain's subdomains, not the old, in order for this to work.

This was suggested here and I thought this was tested, but perhaps the instructions were not clear enough. In any case, I am glad you sorted it out :) and for us it removed the "mistery" on why it did not work.

BTW, here is the explanation on why in your case you needed these redirect lines in .htaccess in the NEW domain folder on the server:

- You have an URL
- You have folders on the server
- You can "point" the domain/subdomain part (host part) of the URL to any folder on a server.
- Wherever it is pointed, this becomes the "root" folder for this domain/subdomain.

This "root" folder does not need to be called the same as the subdomain/domain part of URL (that is, the folder on the server does not need to be called the same as the "host name" part of URL).

In fact, you can point many different subdomains/domains to the same physical folder on the server. It is a configuration where the server is told: When you get a request for this domain/subdomain, go to THAT place on the server.

So in your case, your hosting company has "pointed" both, the old domain and the new domain to the same physical folder on the server.

This is not at all unusual. And which pretty much means you can probably get rid of old subdomain folders on the server.

[Seawitch]

that would be good. I'll ask 'em. I very likely missed the suggestion, due to sheer frustration at how things weren't working.