Redirect old pages to new pages?

I have directories with upper case letters and 80 pages with underscore in

[web2008]

I have directories with upper case letters and 80 pages with underscore in name. For better SERP would like to use lower case letters for directories and hyphens for pages names.

From: www.domain.com/OldFolder/file_name.htm
To: www.domain.com/new-folder/file-name.html

1. What rewrite rule I have to use exactly? (Apache server)
2. For how long rewrites have to stay on htaccess?
3. How long will take search engines to update new pages?
4. What would happen with site rankings?
5. Should I change file extensions from .htm to .html, too?

[lucy24]

If it's OldFolder to new-folder those are entirely different URLs with no way to automate it. Unless you have a lot of names in the form "blahblahfolder" with a recurring element; then there might be a pattern.

1. What rewrite rule I have to use exactly? (Apache server)

:: insert boilerplate about Why We Make You Do It Yourself ::

Show us what you've got and we'll help you with details. The big question is just how many pages are involved, and how many permutations of basic patterns such as lowline vs. hyphen. This will determine whether it's better to keep it all in htaccess, or to rewrite everything to a php page that does the lookups and transformations and then issues a 301 of its own.

2. For how long rewrites have to stay on htaccess?

Do you mean, specifically, 301 redirects? Forever. The day after you delete a redirect on the grounds that nobody has asked for the old URL in many months, some major search engine will stop by for its annual Broad Sweep.

3. How long will take search engines to update new pages?

That depends entirely on your site. Here on WebmasterWorld, new posts are crawled and indexed faster than the page can refresh. On sites like mine, it might take a week for them even to notice a changed URL, and then further days for the indexing to disseminate through all of google's datacenters. If the search engine randomly discovers a number of changed URLs, it might end up re-crawling the whole site to see what's up. (It might also ask for some made-up URLs to check for "soft 404s"; generally this is nothing to worry about.)

4. What would happen with site rankings?

Don't look at me ;)

5. Should I change file extensions from .htm to .html, too?

I have never heard anyone, anywhere, say that one is preferable to the other. If you simply like "html" better, then you might as well use it in the new URLs since you're changing anyway. But otherwise leave things as they are.

[JD_Toims]

What Lucy24 said with a couple of additions/notes.

If it's OldFolder to new-folder those are entirely different URLs with no way to automate it.

FYP: If it's OldFolder to new-folder those are entirely different URLs with no way to automate it in the .htaccess, but you could use PHP ;)

If you simply like "html" better, then you might as well use it in the new URLs since you're changing anyway. But otherwise leave things as they are.

Or strip the stinking extension so it's not necessary to redirect in the future if the underlying technology creating the site changes -- The extension of a URL should have no impact at all on rankings.

4. What would happen with site rankings?

Best case: it won't have much of a direct impact at all.

[phranque]

1. What rewrite rule I have to use exactly? (Apache server)

If it's OldFolder to new-folder those are entirely different URLs with no way to automate it.

FYP: If it's OldFolder to new-folder those are entirely different URLs with no way to automate it in the .htaccess, but you could use PHP ;)

actually it's possible to automate this with mod_rewrite, but i wouldn't bother trying - i would use a scripting language instead.

to get an idea of how, read A guide to fixing duplicate content & URL issues on Apache - Apache Web Server forum:
http://www.webmasterworld.com/apache/3208525.htm [webmasterworld.com]

2. For how long rewrites have to stay on htaccess?

"forever"

3. How long will take search engines to update new pages?

usually weeks or months.
how long does it normally take google to crawl and index pages on your site?
(all of them.)
and then there will be a period of readjustment.

4. What would happen with site rankings?

i would only make these url changes if you were already making a structural/sitewide url change anyway.
if the keywords in urls are already highlighting in search snippets, how much more can you tell google by changing case and using word separators?

5. Should I change file extensions from .htm to .html, too?

If you simply like "html" better, then you might as well use it in the new URLs since you're changing anyway. But otherwise leave things as they are.

Or strip the stinking extension so it's not necessary to redirect in the future if the underlying technology creating the site changes -- The extension of a URL should have no impact at all on rankings.

i would go extensionless instead of tweaking the extension - why expose, promote or commit to your technology?

[lucy24]

Now and then you run into users like me who react to extensionless URLs by shuddering lightly and beseeching the URL to go back in the house and put some clothes on. At most, I'll rewrite an html URL to a php physical page. Why make it obvious to the robots that there's more than meets the eye?

The "OldFolder" vs. "new-folder" question really depends on what the words mean. I took them as lexically distinct names, where "Old" != "new". Even if the basic pattern is simply

([A-Z][a-z]+)([A-Z][a-z]+)
>>
{lower-case of $1}-{lower-case of $2}

you need to consider how many diffent values each [A-Z] can have. If there are more than a few possibilities, it can't be done in htaccess alone unless the config file has the case-changing RewriteMap ("tolower") enabled and declared, and you know what name it uses. Or your name is jdMorgan and you're prepared to slather on the [S] and [N] flags far more liberally than most of us would dare. (A while back I asked my own host and learned that they simply don't have the RewriteMaps enabled on their shared servers. I'd have to go to a VPS, and in my case there's absolutely no reason to.)

[JD_Toims]

Why make it obvious to the robots that there's more than meets the eye?

Because they don't care.

Also, unless you're serving full static headers via Apache hack or through the server-side scripting language, including last-modified [based on the time the page actually changed, which will not be "right now" [likely ever, but] especially for every single visit], [accurate] content-length, 304 Not Modified based on incoming if-modified-since / if-none-match, etc. they'll be able to tell easily it's dynamic even if the extension says HTML, because there are some tell-tale headers and responses not served by standard Apache installations missing for dynamically generated pages, even if the requested URL ended in HTML.

actually it's possible to automate this with mod_rewrite

Okay, okay, okay, you're right, it is technically possible, but you have to check and loop and check and ugh and get headaches, so I don't want to think all that through when I can easily preg_replace() the cap(s) with cap-, str_replace() the _ with -, strtolower() the whole bleeping thing and move on [lol]

[lucy24]

:: detour to LiveHeaders ::

Exhibit A:

HTTP/1.1 200 OK
Date: Mon, 16 Dec 2013 09:55:22 GMT
Server: Apache
Cache-Control: max-age=604800
Expires: Mon, 23 Dec 2013 09:55:22 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1065
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: text/html

Exhibit B:

HTTP/1.1 200 OK
Date: Mon, 16 Dec 2013 09:55:28 GMT
Server: Apache
Accept-Ranges: bytes
Cache-Control: max-age=604800
Expires: Mon, 23 Dec 2013 09:55:28 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1420
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8

What does Accept-Ranges mean?

[web2008]

If it's OldFolder to new-folder those are entirely different URLs with no way to automate it. Unless you have a lot of names in the form "blahblahfolder" with a recurring element; then there might be a pattern.

Show us what you've got and we'll help you with details. The big question is just how many pages are involved, and how many permutations of basic patterns such as lowline vs. hyphen. This will determine whether it's better to keep it all in htaccess, or to rewrite everything to a php page that does the lookups and transformations and then issues a 301 of its own.

I have five different folders with upper case letters.

Folder ServiceBasics would change to service-basics. Folder Archive to archive, Company to company, ServiceTools to service-tools, PDF to pdf etc

Under each folder there are like 10-15 pages.

Is it the same changing url (path to page) and page name for SEO?

www.domain.com/ServiceBasics/oldpagename.htm
www.domain.com/service-basics/oldpagename.htm
vs.
www.domain.com/service-basics/new-page-name.htm

What would be better for SERP - changing just folder upper to lower case or changing the page name too?

[JD_Toims]

What would be better for SERP - changing just folder upper to lower case or changing the page name too?

None of the above -- You're not going to rank better due to this change.

Personally, if I did it [I'm not sure I would], I would do it for URL readability, making sure links aren't accidentally made with a [space] rather than a _ and to get the extensions off the site, because the changes you're wanting to make aren't going to make a ranking difference.

If your URLs were: /123/xyb and your page was about fluffy cats, you'd probably do a bit better with /cats/fluffy, but even then it's only going to help you if you're so far out of the top 10 the few places you might gain aren't going to make any traffic difference.

---

What does Accept-Ranges mean?

It means the client can request only a portion of a document rather than the whole document -- EG a single page of a PDF rather than the entire PDF file.

---

Interesting about some of the headers there -- I haven't seen some of them on any of the hosts I've used unless they're set manually in the PHP file -- Namely, expires + content-length, and sometimes cache-control. The content length being there is likely because the gzip is running after the PHP has generated the content and generates a content-length header.

Did you make those requests with an empty cache?

And which headers are you setting rather than a standard "I don't know Apache/htaccess at all" server configuration would and/or are you setting any via PHP?

Also, are you running a php.ini with expose PHP turned off to get rid of the X-Powered-By purposely?

BTW: You're still missing a last-modified date and/or eTag for if-modified-since or if-none-match headers to be compared to for a 304 response to be served -- Last-modified is usually served as [current time stamp] if it's served for a dynamic page.

---

If you go here: [freetools.webmasterworld.com...] and run it on this page [meaning the URL in the address bar right now], you'll see a much more standard response for a dynamic page with a static extension.

[lucy24]

It means the client can request only a portion of a document rather than the whole document -- EG a single page of a PDF rather than the entire PDF file.

I thought that was part of the request header. Someone-- possibly even you-- explained it to me pretty recently. It's why large PDFs often come through as a series of 206s.

Interesting about some of the headers there -- I haven't seen some of them on any of the hosts I've used unless they're set manually in the PHP file -- Namely, expires + content-length, and sometimes cache-control. The content length being there is likely because the gzip is running after the PHP has generated the content and generates a content-length header.

They're all host defaults. The "Accept-Ranges" header was mystifying because it's only sent with "real" files (I tried other page pairs as well). I don't have any "Header Set" directives, but the "Cache-Control" matches my Expires setting (one week, in seconds).

For comparison purposes, request headers from the same session look like this:

Host: www.example.com
User-Agent: {Firefox on my mac}
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: iu,en-us;q=0.8,en;q=0.6,no;q=0.4,de-de;q=0.2 {these are my OS settings, not specific to the browser}
Accept-Encoding: gzip, deflate
Cookie: {piwik cookie here}
Connection: keep-alive

I didn't bother about clearing the cache, because I don't normally use Firefox, especially not for my own site. There may be some cached content though; I had to refresh some pages to update the CSS.

Also, are you running a php.ini with expose PHP turned off to get rid of the X-Powered-By purposely?

Shared hosting-- I'm not doing anything purposely! What's "X-Powered-By"? Doesn't X- generally mean non-standard/third-party?

[web2008]

I have changed uppercase with lowercase folder name in URL and I am getting page with no errors. But server change smallcase to uppercase.

When I type:
www.domain.com/folder/page.htm
I got the page but it changes to www.domain.com/Folder/page.htm

Since I do not need to test luck with Google and change filenames – lets do only folder change from upper to lowercase letter.

Lucy said that this can be easily done with five separate conditionless RewriteRules.

Lusy please advise.

[phranque]

What does Accept-Ranges mean?

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.5

14.5 Accept-Ranges

The Accept-Ranges response-header field allows the server to indicate its acceptance of range requests for a resource:

Accept-Ranges = "Accept-Ranges" ":" acceptable-ranges
acceptable-ranges = 1#range-unit | "none"

Origin servers that accept byte-range requests MAY send

Accept-Ranges: bytes

but are not required to do so. Clients MAY generate byte-range requests without having received this header for the resource involved. Range units are defined in section 3.12.

Servers that do not accept any kind of range request for a resource MAY send

Accept-Ranges: none

to advise the client not to attempt a range request.

Is it the same changing url (path to page) and page name for SEO?

there are many ways to do it depending on the specifics, but worst case you have several dozen one-to-one redirects.

What would be better for SERP - changing just folder upper to lower case or changing the page name too?

don't do it for the SERP and if you find a good reason to change the urls change it all, not just path or filename.
and get rid of the file extension if you're changing urls.

Namely, expires + content-length, and sometimes cache-control.

http://httpd.apache.org/docs/current/mod/mod_expires.html [httpd.apache.org]:

This module controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses.

i believe the Content_Length header is typically sent by default when apache serves a static file, although this behavior can be disabled.

When I type: www.domain.com/folder/page.htm I got the page but it changes to www.domain.com/Folder/page.htm

what status code did you get for that redirect?

Since I do not need to test luck with Google and change filenames – lets do only folder change from upper to lowercase letter.

if you change any one character of the url, whether it's a "folder" or filename, it's a different url - so you are already testing your luck.

[lucy24]

Wait, stop, rewind.

What is the actual name of your physical file or directory? Some server setups will adjust capitalization to match available files.

If you're changing the capitalization of the URLs you also need to do one of two things: either rename the physical files, or silently rewrite from new URL to old filename.

Worst-case scenario: your server is either case-insensitive or you've got mod_speling [sic] active. This can lead to duplicate content, where "duplicate" = 2^length-of-filename.

Find out what's going on and report back.

[JD_Toims]

i believe the Content_Length header is typically sent by default when apache serves a static file, although this behavior can be disabled.

True, afaik too -- But it's not usually sent for a dynamic file, regardless of the extension, which is why I'm thinking the gzip "kicking in" after the creation of the page by PHP is what's setting the content-length header.

[phranque]

mod_speling would return either a 404 or a 30X.

[lucy24]

mod_speling would return either a 404 or a 30X.

:: detour to Apache docs ::

Whoops! This sounds like something I used to know and have forgotten (emphasis mine):

[If] only one document is found that "almost" matches the request, then it is returned in the form of a redirection response.

Doesn't say what kind of redirect. The only other server-generated redirect I can think of is the directory-slash one from mod_dir, which is a 301.

So that makes one more question for web2008: Does the change in browser's address bar reflect a response to a redirect header? There's no point in redirecting to a different casing if the server is going to redirect right back to another form. In fact that can't be all that's going on, because you would then have an infinite-redirect loop.

[g1smd]

Internally rewrite external requests for URLs with either an upper case character or an underscore to a PHP script that builds the new URL and then sends a 301 redirect using the PHP HEADER directive.

On your non-www to www redirect in htaccess exclude upper case and underscore requests from being redirected by using a preceding RewriteCond.

[lucy24]

On your non-www to www redirect in htaccess exclude upper case and underscore requests from being redirected by using a preceding RewriteCond.

There wouldn't be any uppercase/underscore requests by then. If the end goal of your internal rewrite is to create an external redirect, you'd position the RewriteRule among the ones with [R] flag, not later on among the pure [L] rewrites.

At least, er, ahem, that's how I would do it. (I don't currently have anything with this pattern. But I do have at least one 301 rule that's located among the access-control rules. It redirects to the old-browsers page, making it functionally a 403.)

[web2008]

Find out what's going on and report back.

Lucy sorry for not being clear and thank you for help.

1. Actual name of physical directory is /ToolService/
2. I want to rename physical directory to /toolservice/ as my goal is to keep all in lowercase letters
3. Once I rename physical directory do I need rewrite in .htaccess for search engines to change /ToolService/ to /toolservice/
4. Shared host is 1&1 Apache server.
5. I have 5 different directories that I want to rename to lowercase letters.
6. While testing server when typing /toolservice/ I get /ToolService/ (that is actual correct physical directory name)
8. Now, for testing purposes I renamed physical directory /ToolService/ to /toolservice/ . Upon entering /ToolService/ url in browser – I got /toolservice/ url – with no problems.

What does this mean? Do I just have to rename physical directories to lowercase and not need any rewrite in .htaccess – and site inbound links with uppercase directories will be handled fine. Does it make sense?

9. Current .htaccess

# redirect only example.com to www.example.com. Keep subdomain as it is.
RewriteEngine On
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule ^(.*) http://www.example.com/$1 [L,R=301]

# parse php in html PHP 5.4
AddHandler x-mapp-php6 .html .htm

[edited by: phranque at 4:48 pm (utc) on Dec 17, 2013]
[edit reason] Please Use example.com [webmasterworld.com] [/edit]

[web2008]

To be more clear – when I have renamed physical directory /ToolService/ to /toolservice /

Once I typed /ToolService/ url in browser – server changed /ToolService/ to /toolservice/ in browser bar and I got physical directory /toolservice/ and page with no problems.

Will search engines handle correctly site inbound links with uppercase directories – once I just rename 5 other physical directories to lowercase letters – and do nothing in .htaccess?

[g1smd]

@Lucy Even if the internal rewrite to PHP script is the very first rule, you can still run into problems. Once the rule pattern for the internal rewrite is matched, the htaccess is re-parsed for further matches. Therefore a non-www request for the old URL will often result in the internal PHP script path being exposed as an external www URL. The RewriteCond on the non-www to www redirect is essential to stop that happening.

[phranque]

Once I typed /ToolService/ url in browser – server changed /ToolService/ to /toolservice/ in browser bar and I got physical directory /toolservice/ and page with no problems.

Will search engines handle correctly site inbound links with uppercase directories – once I just rename 5 other physical directories to lowercase letters – and do nothing in .htaccess?

what status code did you get for that redirect?

[web2008]

Once I typed /ToolService/ url in browser – server changed /ToolService/ to /toolservice/ in browser bar and I got physical directory /toolservice/ and page with no problems.

Will search engines handle correctly site inbound links with uppercase directories – once I just rename 5 other physical directories to lowercase letters – and do nothing in .htaccess?

what status code did you get for that redirect?

I didn't get any status code. Just normal page.

[g1smd]

Use the HTTP Live Headers extension for Firefox to examine the HTTP status code returned in the redirect.

[lucy24]

I didn't get any status code. Just normal page.

If the code is 30x you won't see anything onscreen, but look up and you'll see the address bar showing something different from what you typed or clicked.

The case-changing sounds like mod_speling at work. No matter what your htaccess says, mod_speling will check any potential 404s against physical files. And if something matches case-insensitively, it will issue a redirect. In theory this means that you don't need a rule at all, so long as mod_speling is there. But really you should disable the mod and take charge of your own redirecting.

:: detour to re-check ::

Didn't look too closely, but 2.2 and 2.4 seem to be the same.

CheckSpelling Off

This* is the default setting, which your host seems to have overridden. Put this line in your htaccess and it will be as if the module doesn't exist.

Once the module is enabled, a second option is

CheckCaseOnly on

where, again, "Off" is the default. (Meaning that if the mod is enabled at all, it will check everything.)

mod_speling only picks up one misspelling per request. There's no quota on case mismatches; it's simply case-insensitive.


* Just to confuse us, the Apache docs consistently say "on|off" (lower case) in one line and "Off" (Title Case) in the next.

[web2008]

Use the HTTP Live Headers extension for Firefox to examine the HTTP status code returned in the redirect.

Real folder name:
/Test/test.html
Entered in browser lowercase folder name
/test/test.html
Browser changed lowercase to upper case and received page OK.

Results:
----------------------------------------------------------
[domain.com...]

GET /test/test.html HTTP/1.1
Host: www.domain.com
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate

DNT: 1
Connection: keep-alive

HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Dec 2013 23:23:20 GMT
Server: Apache
Location: [domain.com...]
Content-Length: 257
Keep-Alive: timeout=2, max=200
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
----------------------------------------------------------
[domain.com...]

GET /Test/test.html HTTP/1.1
Host: www.domain.com
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate

DNT: 1
Connection: keep-alive

HTTP/1.1 200 OK
Date: Tue, 17 Dec 2013 23:23:20 GMT
Server: Apache
X-Powered-By: PHP/5.4.23
Keep-Alive: timeout=2, max=199
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
----------------------------------------------------------

[JD_Toims]

<SlightlyOT>
Sorry Lucy, I haven't had time to make it back up this thread to where we were talking about headers relating to "standard HTML" v "HTML extension rewritten to PHP", but this response is "stock-standard" for an HTML extension rewritten to PHP:

HTTP/1.1 200 OK
Date: Tue, 17 Dec 2013 23:23:20 GMT
Server: Apache
X-Powered-By: PHP/5.4.23
Keep-Alive: timeout=2, max=199
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html

I'm sure you'll notice the *huge* difference in what you're seeing from your site/host compared to the above.
</SlightlyOT>

[web2008]

Sorry admin - I didn't know that I need to use example.com

Real folder name
http://www.example.com/Archive/test.html


Lower case folder name entered
http://www.example.com/archive/test.html


Results:
-------------------------------------------
http://www.example.com/archive/test.html

GET /archive/test.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive

HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Dec 2013 23:43:27 GMT
Server: Apache
Location: http://www.example.com/Archive/test.html
Content-Length: 260
Keep-Alive: timeout=2, max=200
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
----------------------------------------------------------
http://www.example.com/Archive/test.html

GET /Archive/test.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive

HTTP/1.1 200 OK
Date: Tue, 17 Dec 2013 23:43:27 GMT
Server: Apache
X-Powered-By: PHP/5.4.23
Keep-Alive: timeout=2, max=199
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
----------------------------------------------------------

[lucy24]

web2008, see earlier post about mod_speling.

:: wondering how much longer this can go on before phranque puts his foot down and starts chopping ::

this response is "stock-standard" for an HTML extension rewritten to PHP:

The quoted response is identical to what Live Headers gives me when I try the same request in MAMP. (Right down to php version: You didn't get it from your own MAMP did you?) That's for any request, whether html physical file or html rewritten to php.

I'd assumed the "cache-control" and "expired" lines were because I have explicit caching times set in htaccess, and the numbers match. But I have the same lines in my local file's htaccess, and the headers are absent.

[JD_Toims]

The quoted response is identical to what Live Headers gives me when I try the same request in MAMP. (Right down to php version: You didn't get it from your own MAMP did you?) That's for any request, whether html physical file or html rewritten to php.

Nope, it came from web2008's LiveHeaders example above, meaning that's the response returned by a "live" server -- Have a look at the last header-set in the post preceding yours.

But I have the same lines in my local file's htaccess, and the headers are absent.

That's what I've seen previously on "live" servers too, in fact, yours is the first I've seen send both those two via .htaccess rather than having to set them in PHP -- I think I've seen cache-control sent by one server before not expires or content length -- Note: I wasn't gzipping when I was testing, so content-length may be more "normal" to be sent without manipulating anything in that case.