Rewrite Rule for directory content being moved to two places

ecommerce website

[leon_di]

Hi All

I've been reading the forum with interest with regards to rewrite rules but didn't come across anything that covered my scenario so I help someone can help.

We're currently restructuring our ecommerce website and as a result two things are happening:

example.com/folder1/subfolder1 is moving to example/folder2/subfolder1

This is correct for category and subcategory pages but not product pages as below:

example.com/folder1/product1.html is moving to example.com/product1.html

Will the following work:

RewriteCond %{HTTP_HOST} example.com/folder1/$ [NC]
RewriteCond %{HTTP_HOST} !subfolder1 [OR]
RewriteCond %{HTTP_HOST} !subfolder2 [OR]
RewriteRule ^folder1/(.*)$ /$1 [R=301,NC,L]
RewriteRule ^folder1/(.*)$ /folder2/$1 [R=301,NC,L]


or will I need to replace the last rule with:

RewriteRule ^folder1/subfolder1/(.*)$ /folder2/subfolder1/$1 [R=301,NC,L]

Thanks in advance for your help!

[lucy24]

Before anything else: leave a blank line after each RewriteRule. This is for your own sake, not the server's. It's especially important when one rule has Conditions, so you remember that the conditions only apply to the first rule.

RewriteCond %{HTTP_HOST} example.com/folder1/$ [NC]

This is flat-out wrong. The "host" is the (sub)domain only, i.e. the part before the first slash. So this condition will always fail, and the first rule will never execute. (The second rule has no conditions.)

Never use [NC] in anything referring to your own site. There's only one possible correct casing; use it and save the server some work.

RewriteCond %{HTTP_HOST} !subfolder1 [OR]
RewriteCond %{HTTP_HOST} !subfolder2 [OR]

This is also wrong, but for different reasons. Here you don't mean host at all; you mean %{REQUEST_URI}. Furthermore, the conditions as written don't make sense: the pair says "request is not A and/or not B"-- a condition that will always succeed. You probably meant to say "request is not A and not B"

Since both conditions are looking at the same variable, it's more efficient to pipe them:

RewriteCond %{REQUEST_URI} !(subfolder1|subfolder2)

... which concurrently converts two lines that don't make sense into a single line that does: "request is not for A or B".

In fact I had expected the second [OR] to be a lethal error ("either this condition or the next" -- but there is no "next") and can't quite figure out why it isn't. (I experimented.) But never mind that.

It sounds as if what you're really aiming for is "move all subfolders from this old folder to this new one". If it's meant to apply only to material in subfolders (product pages aren't in subfolders) the rule is best expressed as a conditionless

RewriteRule ^folder1/([^/]+/.*) http://www.example.com/folder2/$1 [R=301,L]

Again, no need for [NC] -- but do include full protocol-plus-domain in your target.

[leon_di]

Hi Lucy

Thanks so much for taking the time to reply so comprehensively and showing me the errors of my thinking.

You've interpreted what I'm trying to achieve correctly and thanks for that. There is a second part to the issue, which is to move the product pages to the root. What would be the best way to do this, given the RewriteRule you've given above?

[phranque]

welcome to WebmasterWorld, leon_di!

what do your exemplified mod_rewrite directives look like after the changes?
have you cleared your browser cache before testing the new redirects?

[lucy24]

example.com/folder1/product1.html is moving to example.com/product1.html

Does /folder1/ contain any pages that are not product pages?

[leon_di]

Yes, it does contain pages that are not product pages.

Hi phranque, we've not run tests yet. Planning this for next week. Trying to prepare the htacess file for when we do.

[lucy24]

Yes, it does contain pages that are not product pages.

I should have worded my question more narrowly: Of course it contains non-product pages, since that's the whole point of the redirect. But do any of them occur at the top level of the folder, like

example.com/folder1/non-product-page.html

Are there lots and lots of product pages and just a few non-products in this location? Are the URLs distinguishable in any way? For example, maybe all product pages end in a number, while non-product pages don't. Or all product pages contain some particular bit of text. Even something as simple as "product page names contain a hyphen while non-product pages don't".

[leon_di]

Hi Lucy

Yes, we have pages that are structured as:

example.com/folder1/delivery

They don't have a file extension at the end e.g. .html, whereas the products do.

Thanks so much for your help again.

[lucy24]

example.com/folder1/product1.html is moving to example.com/product1.html
...
They don't have a file extension at the end e.g. .html, whereas the products do.

You're now looking at two separate rules, one for product pages and one for the /folder1/subfolder/ redirect:

RewriteRule ^folder1/([^./]+\.html) http://www.example.com/$1 [R=301,L]

RewriteRule ^folder1/([^/]+/.*) http://www.example.com/folder2/$1 [R=301,L] 

Since the product-page URLs and the folder1/subfolder/ URLs are mutually exclusive, it makes no functional difference which order you put these two rules in. Instead, list them in order of likelihood, based on your own knowledge of the site.

That's assuming your product pages will retain their .html extension. If you want to remove it at the same time, simply shift the capturing parenthesis so the pattern is

^folder1/([^./]+)\.html

Depending on the exact form of your product page names, you might replace [^./]+ with \w+ or [a-z]+ or [a-z]+-\d+ ... or whatever gives the most exact match. (I'm just making these up at random.) But now we're into fine-tuning territory.

[phranque]

what should the response be for http://www.example.com/folder1/ requests?

[lucy24]

Oh, good question. It looks as if some pages-- the non-product pages with extensionless URLs-- are staying behind in /folder1/. By the existing rules, requests for /folder1/ (the subdirectory index) would also be unchanged. Is that what you intended, leon? Does this URL exist?

Which reminds me: Have you got an index redirect in place? Search engines will ask for /folder1/index.html by name. By the rule intended for product pages, this would lead to an unintentional redirect to the root unless you add an exception to the product-page rule:

RewriteCond %{REQUEST_URI} !index\.html

This is only necessary if the product-page URL is expressed in the rule as [^./]+ or \w+. If you end up using a tighter pattern, "index.html" might not fit anyway.

[leon_di]

Hi Lucy/Phranque

To answer the questions in your first response:

The product pages will retain their .html extensions.

Our product page names are quite simply brandname-modelname.html. Some model names include numbers, others don't.

I've just noticed that the structure of our category pages will cause an issue with the product rule proposed. Unfortunately, category pages are set to use the .html extension (should not have been but are). Therefore, we have category pages following this format: example.com/folder1/category-name.html and these will be moving to example.com/folder2/category-name.html. These will fall under the product rule and be redirected to example.com/category-name.html.

In response to your second post:

The non-product pages are not staying in /folder1. They will also be moving to the root. Sorry if I didn't make this clear. So it will follow the format of example.com/delivery (with no file extension) instead of example.com/folder1/delivery. Therefore, the example.com/folder1/ url will no longer exist.

Yes we do have an index redirect in place.

Thanks again for your help.

[phranque]

RewriteRule ^folder1/$ - [G]

[lucy24]

Unfortunately, category pages are set to use the .html extension (should not have been but are). Therefore, we have category pages following this format: example.com/folder1/category-name.html and these will be moving to example.com/folder2/category-name.html. These will fall under the product rule and be redirected to example.com/category-name.html.

At this point, a php script begins to look good :)

How many category pages are there? If it's a manageable number-- let's say no more than about ten-- make a rule that lists them by name, and put this rule before the product-page rule.

The non-product pages are not staying in /folder1. They will also be moving to the root. Sorry if I didn't make this clear.

So everything except subfolders and categories is moving to the root? Then the pattern for the product-page rule isn't

^folder1/[^./]+\.html

but simply

^folder1/[^/]+$

If the entire /folder1/ directory is going away, then redirecting explicit requests for
/folder1/index.html
to
/index.html
is actually perfectly reasonable. Nobody will make these requests except search engines. Redirecting one page to the root is fine. They only get agitated when you redirect everything to the root.

Once you've set up all redirects, leave the physical /folder1/ directory alone for a while. After a week or two, check logs and make sure that all requests for this directory get a 301 response. (Ordinary access logs won't say what you're redirecting to; they'll just note the 301.) Once you've verified this, you can delete the physical directory.

That's assuming, of course, that all these redirects correspond to physical directory changes.

[leon_di]

This was my original thinking - to set a rule to redirect all pages to the root except all category pages specified (with poor application). There are less than 10 categories so it is manageable.

[lucy24]

So you've got three rules now.

Rule 1 (this has to come first) for category pages by name, here assuming they end in .html and will continue to do so:

RewriteRule ^folder1/(file|otherfile|thirdfile|fourthfile)\.html http://www.example.com/folder2/$1.html [R=301,L]

Rule 2 for all other pages at the top level of folder1:

RewriteRule ^folder1/([^/]+)$ http://www.example.com/$1 [R=301,L]

Rule 3 for all remaining material in folder1, i.e. stuff in subfolders:

RewriteRule ^folder1/(.*) http://www.example.com/folder2/$1 [R=301,L]

That's if the entire content of /folder1/ is to be removed, with part of it going to the root and the rest-- including all subfolders-- going to /folder2/

Note that with this new ruleset, the former "pattern" for the subdirectory rule
^folder1/([^/]+/.*)

has been simplified to
^folder1/(.*)

because you're simply scooping up everything left over after the first two rules have executed.

---

Do you care what happens when search engines ask for
/folder1/subdirectory
without trailing slash? (They will do this just to test you, in the same way that they ask for /index.html in all known directories.) By default, mod_dir issued a 301 redirect to
/folder1/subdirectory/
but this will now be overridden by rule 2, creating a redirect instead to
/folder2/subdirectory
which mod_dir will then redirect (correctly) to
/folder2/subdirectory/
Ordinarily you shouldn't have a double redirect, but since this one is caused purely by the search engine's own behavior, you may decide it isn't worth further fine-tuning.

[phranque]

So you've got three rules now.

the example.com/folder1/ url will no longer exist

4 rules...

[lucy24]

4 rules...

If every URL in /folder1/ receives a redirect of some kind, there's nothing left to get the 410, unless you prepend a rule #0

RewriteRule ^folder1(/(index\.html)?)?$ - [G]

This might be construed as misleading, since it makes it look as if the entire folder is gone rather than relocated.

Come to think of it, did we ever talk about the actual index page of /folder1/, assuming it exists? What happens to it? That is: what happens to its content, or to the information the page currently provides?

Is /folder2/ already present, or is it newly created?

[leon_di]

/folder2/ is already present.

There is content on /folder1/ but this will be combined into the index page for folder2/

When search engines ask for /folder1/subdirectory, will the page not be redirected to /folder1/subdirectory.html. As explained previously, our category pages are set to have the file extension .html added. Ideally we should change this however to be subdirectory/ .

[lucy24]

Before anything else:
the bottom part of post 4729971 (from me) is wrong. I said "redirect to /folder2/" when in fact this material will redirect to the root. So the existing rule will not end up doing what you wanted it to do. (Normally I'd ask phranque or someone like him to fix my post, but here there's been follow-up discussion so it would get too messy.)

Quick lesson on mod_dir:
Suppose you have a real, physical directory

/folder1/subdirectory/

Whenever someone asks for this URL, mod_dir will kick in (using the DirectoryIndex directive) and show the page

/folder1/subdirectory/index.html

without changing the browser's address bar. You can think of it as a special type of rewrite.

But also
If someone asks for

/folder1/subdirectory

without trailing slash, then if /subdirectory/ is a real, physical directory, mod_dir will issue a 301 redirect to

/folder1/subdirectory/

With me so far?

Enter the search engines. Once they know that

/folder1/subdirectory/

is a valid URL, they will periodically ask for two other things:

/folder1/subdirectory

and

/folder1/subdirectory/index.html

The first will be redirected by mod_dir. The second will be redirected by the index redirect that you've explicitly put in your htaccess: generally it's the second-to-last 301 redirect, immediately before the domain-name-canonicalization redirect.

If you want to be nasty, you can think of it as entrapment. Search engines are explicitly looking for Duplicate Content (identical material available at more than one URL).

All of this happens after the initial request has passed through mod_rewrite (and most other mods, depending on setup); mod_dir is basically a fallback.

:: pause here to paste the above into /boilerplate/ directory in case I ever want to reuse it ::

Now, what happens on your site, where you've got explicitly coded redirects?

Requests for

/folder1/subdirectory

without trailing slash will be picked up by the current Rule 2, which is intended to handle requests for anything inside /folder1/ that is not one of the short list from Rule 1, and also not in a subdirectory (because the request doesn't have the final slash). In your particular case, this will result in the material being sent to the root.

There are two ways to handle this. The messy way is

RewriteCond %{REQUEST_URI} !-d

meaning "only do this if the requested file isn't a real, physical directory".* But that requires the server to do an additional lookup. If /folder1/ contains a reasonable number of subdirectories, it's better to list them by name:

RewriteCond %{REQUEST_URI} !/folder1/(subdir|othersub|thirdsub)$

At this point you might as well throw out the Condition and instead put in a supplementary Rule 1b to go before the existing Rule 2:

RewriteRule ^folder1/(subdir|othersub|thirdsub)(/index\.html)?$ http://www.example.com/folder2/$1/ [R=301,L]

See the part with the ? question mark? You've now killing two birds with one stone by intercepting both forms of the malformed /folder1/subdirectory/ request. The correctly formed request, with final slash and that's all, continues to be handled by Rule 3.

Heh, phranque, you're right. Four rules :)

:: hoping that this gets sorted out before the rulecount reaches 34, haha ::


* Here I'm assuming that the -d test applies even if the request happens not to end in a slash (yet). If it doesn't, the test is worthless.

[lucy24]

Separate post because it was getting too long:

As explained previously, our category pages are set to have the file extension .html added. Ideally we should change this however to be subdirectory/

I wish you wouldn't. A spurious / final slash when you've also got real, physical directories lying around creates a mess. On the other hand, changing the physical pages from

/folder1/category

(presumably rewritten to /folder1/category.xtn)
to

/folder2/category/index.html

(with URL simply /folder2/category/)
would be perfectly reasonable if the new /category/ directories had other content as well. Otherwise it's kinda pointless.

The good news is that if category pages are the only extensionless material in the top level of /folder1/ then the subdirectory rule can be simplified:

Rule 1: category pages:

RewriteRule ^folder1/(file|otherfile|thirdfile|fourthfile)$ http://www.example.com/folder2/$1.html [R=301,L]

Rule 2 (rule 1b from preceding post) for real subdirectories:

RewriteRule ^folder1/([^./]+)(/index\.html)?$ http://www.example.com/folder2/$1/ [R=301,L]

Rule 3 (former Rule 2) for all other pages at the top level of folder1, i.e. anything with an extension:

RewriteRule ^folder1/([^/]+)$ http://www.example.com/$1 [R=301,L]

Rule 4 (former Rule 3) for all remaining material in folder1, i.e. stuff in subfolders:

RewriteRule ^folder1/(.*) http://www.example.com/folder2/$1 [R=301,L]

Have I got that right?

Please say you don't have any subdirectories within /folder1/ whose name contains a literal . because then things get complicated again.

[leon_di]

Hi Lucy

Thanks for explaining mod_dir and what it does.

If keeping the .html extension for the categories keeps things more simple, we may as well keep them.

With regards to your question about names containing a literal, no there isn't (if I understand rightly that you mean characters " ' ").

Thanks again for your help. Should be able to start testing this over the next few days.

[lucy24]

if I understand rightly that you mean characters " ' "

In this case, the specific character . (dot, period, full stop). It is perfectly legal to use a dot in directory names-- witness apache dot org, which has directories for /2.2/ and /2.4/ --and possibly even in filenames. But it's a heck of a lot simpler if you only use the dot as an extension delimiter.

:: wandering off to deal with unhappy discovery that on my site, the -d and -f tests don't work with %{REQUEST_URI} but only with %{REQUEST_FILENAME} --although the good news is that the final slash is optional for directories ::

[leon_di]

Hi Lucy

No we don't use . in any of the subdirectories.

There are 3 subcategories that will be removed. Is the right thing to do to add a rule for them at the top, so there will be a new rule 1? For example:

example.com/folder1/removed.html (which is a category page) and all subcategories.

RewriteRule ^folder1/removed$ - [G]

and is there a way to combine multiple subdirectories together in one rule as you have in rule 1 above:

RewriteRule ^folder1/(removed1|removed2|removed3)$ - [G]

========================================================

A little more information about the structure of our website:

It's was set up a few years ago as essentially 5 different stores under one domain (using Magento), with each store having its own top level directory. E.g.

example.com/shop1/category1/subcategory1
example.com/shop2/category1/subcategory1 and so on...

We're now restructuring this all to come under 1 store (and reducing our product catalog substantially, eliminating 3 categories altogether) which is why were losing the top level directory.

example.com/category1/subcategory1
example.com/category2/subcategory1

(note 1: as explained before, category pages have been given a .html extension so it looks like example.com/category1.html)
(note 2: the categories are being re-arranged to make sense to the end user so it's not as simple as removing the shop level directories and the rules you've set out above reflect what is happening in 4 of the 5 stores)

folder1 in the rules above relate to the store top level directory.

I therefore, need to repeat these rules for the other 3 stores. What is the most efficient way to do this?

Thanks again for your help.

[leon_di]

Actually, having taken a closer look at the other store urls, the new structure for the other stores differ to the ones we've already worked out.

One is being removed completely so could I use:

RewriteRule ^shop2$ - [G]

The third has 2 top level categories being removed, and another moved into a subcategory of another:

shop3/category1 -> moved to /category1
shop3/category2 -> moved to /category2
shop3/category3 -> moved to /category1/category3
shop3/category4 -> removed
shop3/category5 -> removed

I've worked out the following from the rules we already have (referenced your rules numbers for convenience):

RewriteRule ^shop/(category4|category5)$ - [G]

Rule 1: category pages:
RewriteRule ^shop3/category1$ http://www.example.com/category1/$1.html [R=301,L]

RewriteRule ^shop3/category2$ http://www.example.com/category2/$1.html [R=301,L]

RewriteRule ^shop3/category3$ http://www.example.com/category1/category3$1.html [R=301,L]

Rule 2 (rule 1b from preceding post) for real subdirectories:
RewriteRule ^shop3/category1([^./]+)(/index\.html)?$ http://www.example.com/category1/$1/ [R=301,L]

RewriteRule ^shop3/category2([^./]+)(/index\.html)?$ http://www.example.com/category2/$1/ [R=301,L]

RewriteRule ^shop3/category3([^./]+)(/index\.html)?$ http://www.example.com/category1/category3/$1/ [R=301,L]

Rule 3 (former Rule 2) for all other pages at the top level of folder1, i.e. anything with an extension:
RewriteRule ^folder1/([^/]+)$ http://www.example.com/$1 [R=301,L]

Rule 4 (former Rule 3) for all remaining material in folder1, i.e. stuff in subfolders:
RewriteRule ^shop3/category1(.*) http://www.example.com/category1/$1 [R=301,L]
RewriteRule ^shop3/category2(.*) http://www.example.com/category2/$1 [R=301,L]
RewriteRule ^shop3/category3(.*) http://www.example.com/category1/category3/$1 [R=301,L]

I imagine I'm completely wrong as usual! Seems inefficient with 11 rules to achieve this.

[lucy24]

RewriteRule ^shop2$ - [G]

Double-checking: this is intended only to apply to requests for
example.com/shop2
and not to
example.com/shop2/
or
example.com/shop2.html
or
example.com/shop2/pagename.html
?

and is there a way to combine multiple subdirectories together in one rule as you have in rule 1 above:

RewriteRule ^folder1/(removed1|removed2|removed3)$ - [G]

Yes, you can absolutely use pipes that way in the pattern.

In forms like this:

RewriteRule ^shop3/category1$ http://www.example.com/category1/$1.html [R=301,L]

RewriteRule ^shop3/category2$ http://www.example.com/category2/$1.html [R=301,L]

what does the $1 refer back to? I don't see a capture. I'm not sure, but it's possible you actually meant

RewriteRule ^shop3/(category1|category2)$ http://www.example.com/$1/ [R=301,L]

or maybe

RewriteRule ^shop3/(category1|category2)$ http://www.example.com/$1.html [R=301,L]

In fact it's possible that for some of your redirects you're looking at a generic pattern

RewriteRule ^foldername/(subdir1|subdir2|subdir3)/(.+) http://www.example.com/$1/$2 [R=301,L]

[leon_di]

Hi Lucy

RewriteRule ^shop2$ - [G] is intended to apply to everything that comes after the /shop2 directory, including the examples you provided.

Should it be RewriteRule ^shop2(.*)$ - [G]

I actually meant RewriteRule ^shop3/(category1|category2)$ http://www.example.com/$1.html [R=301,L]. Thanks for correcting me.

The generic pattern can replace which redirects? Do you mean rules 1 & 4?

[lucy24]

Should it be RewriteRule ^shop2(.*)$ - [G]

Since you're not capturing for reuse, it can simply be

RewriteRule ^shop2 - [G]

without closing anchor. This rule has to go after any rules that redirect specific pages or subdirectories within /shop2/ -- like your original situation where some pages move from /folder1/ to /folder2/ while others move to the root. This is an exception to the general "list rules in order of severity" principle.

If the entire content of /shop2/ is moving to various other places, then the only time you'll end up returning the [G] response is when people request the directory itself, most precisely expressed as

^shop2(/(index.html)?)?$

although you won't necessarily have this exact formulation anywhere. It depends on rule ordering: at some point, there has to be a rule for "anything in /shop2/ that hasn't already been covered by some other rule".

Which reminds me...
You may want to have a custom 410 page just for the former example.com/shop2/ (what used to be the directory index page). That's where you tell your human users "We've moved to such-and-such page".

In htaccess, you do it like this: within /shop2/ (the physical directory) put a new little htaccess that consists of the single line

ErrorDocument 410 /shop2/shop2gone.html

using whatever name and path you like. The "shop2gone" document itself does not have to be inside the directory-- but the ErrorDocument directive does. This, in turn, means that you have to keep the physical directory, even if you've moved or deleted all of its former contents.

---

The generic pattern can replace which redirects?

Truthfully I didn't fine-tooth-comb the latest version. But any time you've got a group of rules that say

RewriteRule ^folder/subdir1/(.*) http://www.example.com/subdir1/$1 [R=301,L]

RewriteRule ^folder/subdir2/(.*) http://www.example.com/subdir2/$1 [R=301,L]

RewriteRule ^folder/subdir3/(.*) http://www.example.com/subdir3/$1 [R=301,L]

you could collapse them all to

RewriteRule ^folder/(subdir1|subdir2|subdir3)/(.*) http://www.example.com/$1/$2 [R=301,L]

That's assuming that the "subdir1" etc. names will be the same after the redirect. If you're concurrently renaming in non-predictable ways, they would each have to be a separate redirect.

[leon_di]

If the entire content of /shop2/ is moving

That entire section is essentially being deleted. As such I want to tell the search engines that they are gone.

Truthfully I didn't fine-tooth-comb the latest version.

Do the following look correctly formed?

RewriteRule ^folder/(subdir1|subdir2|subdir2/subsubdir)$ - [G]

RewriteRule ^folder/(subdir1|subdir2)\.html www.example.com/$1/$2.html [R=301,L]

RewriteRule ^folder/(subdir1|subdir2)$([^./]+)(/index\.html)?$ http://www.example.com/$1/$2/ [R=301,L]

RewriteRule ^folder/(subdir1|subdir2)/(.*) http://www.example.com/$1/$2 [R=301,L]

Thanks Lucy. I appreciate your help.

[lucy24]

Do the following look correctly formed?

RewriteRule ^folder/(subdir1|subdir2|subdir2/subsubdir)$ - [G]

The closing anchor means this rule is only intended for requests for:
example.com/folder/subdir1
example.com/folder/subdir2
example.com/folder/subdir2/subsubdir

If that was your intention, it's correct-- except that it can be collapsed to
(subdir1|subdir2(/subsubdir)?)
assuming /subsubdir/ really is a subdirectory of /subdir2/

But if this rule is located anywhere other than the very end of all directs involving /folder/, and you're talking about (former) real, physical directories, the pattern should instead be

^folder/(subdir1|subdir2(/subsubdir)?)(/(index\.html)?)?$

to intercept all possible directory-index requests.

---

Incidentally ... There is a firm rule against naming your own domain. Hence the recurring example.com (.org, .net etcetera). But in general it's perfectly OK to use the actual directory names. Sometimes this makes it easier to visualize what you're doing.