Pulling my hair out over 301 redirect not working

[trentiles]

So I moved a blog from Blogger to self hosted Wordpress. Upon doing that I also changed from bloggers /year/month/title.html url's to /category/title/ urls. While on blogger I used a custom URL which is the same url currently being used. So I never used #*$!.blogspot.com.

Google has all the old Blogger url's indexed and none of the new so inside my .htaccess file I manually entered a 301 redirect for every url. There were around 80 of them.

The problem is when I click on the old url in google it still gives me a 404 error page. BUT if I refresh the page then it shows the correctly redirected url. Shouldn't that happen automatically without the need for a refresh?

I'm so stumped.

Here's my .htaccess file:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

Redirect permanent /2009/01/classic-mexian-breakfast.html http://www.example.com/beans-lentils-legumes/classic-mexican-breakfast/

then around 80 more url redirections.

I've tried both redirect 301 and redirect permanent, both do the same thing.

Please help as I am losing traffic.

Thanks everyone

[edited by: jdMorgan at 1:55 pm (utc) on Feb. 26, 2009]
[edit reason] example.com [/edit]

[jdMorgan]

1) Don't mix mod_alias (e.g Redirect permanent) and mod_rewrite rules. If you use mod_rewrite for any, use it for all. This prevents problems caused by varying execution order of mod_alias and mod_rewrite directives.

2) Put all external redirects first, in order from most-specific (fewest URLs affected) to least-specific (most URLs affected), followed by internal rewrites, again in order from most- to least- specific.

Translated, that means that the WordPress internal rewrite code that you show above must be *after* all your external redirects, and that you should use mod_rewrite to do those external redirects. And taking external redirects as an example, the rules that redirect single URLs should be first, followed by rules that redirect groups of URLs based on common characteristics, and the final redirect will probably be the rule that redirects all non-www domain requests to the "www" domain, which affects *all* URLs in that non-www domain.

[edit] I forgot to mention that the most likely cause of your trouble is that you did not completely-flush your browser cache after changing the code on your server. Your browser was therefore showing you previously-cached 404 responses. [/edit]

Jim

[edited by: jdMorgan at 2:03 pm (utc) on Feb. 26, 2009]

[trentiles]

I will make the changes you specified and see what happens.

Browser cache was cleared and I even had friends try the old URL's (they had never been to any pages at this domain) and they even received the 404's.

I will let you know what happens after I change all the individual redirects to use mod_rewrite.

[trentiles]

After doing some research I'm coming up a bit confused.

So I shouldn't be using Redirect 301 oldurl newurl ?

[jdMorgan]

Please see the documents cited in our Forum Charter, and the tutorials in our Forum Library. Links to both are up near the top of this page.

It's still likely that caching is a problem; Perhaps your friend uses an ISP which routes all users through a caching proxy; The fact that a browser refresh changes anything means that the initial request did not actually reach your server, and that a stale 404 response was displayed. This can be verified by looking at your raw server access logs.

I suggest that you use the external redirect syntax of RewriteRule for your redirects, rather than mixing directives from mod_alias and mod_rewrite. This will allow you to be sure that the redirects are executed first, and avoid the situation where a URL is first rewritten and then gets redirected: If this happens then clients (e.g. browsers and search engine robots) will be able to "see" the internally-rewritten filepath as a URL, potentially leading to search ranking problems, bad bookmarks, and user confusion. This is a secondary issue, not related to the main problem you are reporting here.

Jim

[MadeWillis]

maybe try adding ?clearcache=true to the end of the URL and hit enter?

[jdMorgan]

Note that "clearcache=true" is a completely-arbitrary string; It simply forces the browser and network to do an end-to-end (browser to server) reload, because it appears to be a new and unique URL, and therefore, not servable from any previously-saved copy in the browser or any network cache. If you change the query string for each request, then you won't ever see a cached response.

Jim

[trentiles]

Well I've read all the stuff you recommended and still cannot get things to work. This rewrite stuff is confusing!

For example I need: http://www.example.com/2008/12/pumpkin-spice-dip.html to 301 to http://www.example.com/snacks/pumpkin-spice-dip/

Now on my server /2008/12/articlename.html does not exist..not sure if that matters or not

Here's what I have tried:

RewriteRule ^2008/12/pumpkin-spice-dip\.html$ http://www.example.com/snacks/pumpkin-spice-dip/ [R=301]

All the above does it redirect to http://www.example.com

Sorry I'm such a noob at this.

[edited by: coopster at 5:14 pm (utc) on Feb. 26, 2009]
[edit reason] please use example.com, thanks! [/edit]

[jdMorgan]

You code is correct with the exception of the missing [L] flag:

RewriteRule ^2008/12/pumpkin-spice-dip\.html$ http://www.example.com/snacks/pumpkin-spice-dip/ [R=301[b],L[/b]]

But that is only an (important) code-efficiency consideration; The redirect should still have worked.

I suspect that you are seeing the action of your custom 404 error page being invoked -- if you have one. If not, then perhaps this is your script's response to a missing page. In either case, this would indicate that the rule did not execute.

Did you preface your new redirect rule(s) with the required directives to enable mod_rewrite?

Options +FollowSymLinks
RewriteEngine on
#
RewriteRule ^2008/12/pumpkin-spice-dip\.html$ http://www.example.com/snacks/pumpkin-spice-dip/ [R=301,[b]L[/b]]
RewriteRule ^2009/01/classic-mexian-breakfast\.html$ http://www.example.com/beans-lentils-legumes/classic-mexican-breakfast/ [R=301,L]

You may find it quite helpful to download, install, and learn to use the "Live HTTP Headers" add-on for Firefox and Mozilla-based browsers. This tool will show all HTTP transactions used to serve a requested page, and allow you to check for incorrect or chained multiple redirects on requests. I highly recommend it as part of the basic Webmaster kit.

Jim

[trentiles]

Thanks Jim. It's all working now! I really appreciate it.