How big is too big for an htaccess file?

I have a huge site with 100,000's of pages. htaccess file is over 25 megs.

[acimag]

I know that is not good at all. There are patterns that I might be able to do lets say 25,000 lines at a time but it involves seeing the pattern, matching and then converting to lower case. is that possible. I'll show you 2 lines maybe you can see the pattern and help... I hope.

RewriteRule ^Emergency-Fast-Business-Rental/Hot/Connecticut/Fairfield/Aqua_Vista_Fast_Temp_Quick_Gas_Heaters_Rentals.html http://www.example.com/aqua_vista-ct-temporary_blah_blah_hot_heaters.html


Now this pattern changes the beginning before connecticuit and then after 'Aqua_Vista' and even the state and county change for every listing. I don't want a bad site performance.

[g1smd]

If possible the new URL should use hyphens, not underscores.

This then becomes a simple problem to rewrite (that's rewrite not redirect) all requests with underscores to a PHP script that manipulates the URL and sends the HTTP 301 header and the new URL header OR sends 404 Not Found when there is no new URL to redirect to.

This means the htaccess performance for all requests is quick as the file is short.

The PHP file can be as large or as small as you like, it will only be invoked for requests that need to be redirected.

The PHP file could be as simple as old and new URLs as pairs in an array, or patterns covering multiple URLs in an array, or other logic as required. Whatever you do, Regular Expressions will probably feature quite highly.

[acimag]

So you telling me my htaccess file can be as large as I want like 50 megs and it won't slow down the site?

Thats insane. I thought the file would atleast have to load so all the rewrites would work.

as for everything else you said its way over my head and i've tried reading a lot about this subject but found very limited resources.

[g1smd]

I said the PHP file can be as large as you like.

If you rewrite underscore requests to a PHP script, then that's one line of code in the htaccess file.

RewriteRule _ /handlethis.php [L]

The PHP file does the real work, looking up the new URL that replaces the old, and sending the correct HTTP headers (301 or 404) back to the browser.

You can have thousands of lines in htaccess if you want, but the code becomes unmanageable. There are better solutions.

[acimag]

yeah i just finished putting in all the redirects the file is now 73mgs.

and the site runs VERY slow now. I noticed i do not have the [L] at the end. does that matter.

[acimag]

correction I just added the [L] and is VERY SLOW. any idea's?

[topr8]

>>correction I just added the [L] and is VERY SLOW. any idea's?

of course, 50 megs is way too big for an htaccess file
and you could also have rules in there that are taking a long time (relatively) to process.

[lucy24]

TWENTY-FIVE MEGABYTES?

You did say megs, right, not kilobytes?

That makes this entire thread rhetorical. 25 MB is too big for anything,* let alone an htaccess file.

In particular, keep in mind that the full htaccess is read at every single request, not just pages. Even the [L] flag just means "Don't execute any more mod_rewrite rules this time around", not "Stop reading this file".


* That is: Anything the user opens without advance notice. If you've linked to a 600MB PDF, human decency requires you to say so.

[lucy24]

:: returning after reading first post more slowly ::

There are patterns that I might be able to do lets say 25,000 lines at a time but it involves seeing the pattern, matching and then converting to lower case. is that possible. I'll show you 2 lines maybe you can see the pattern and help.

Unfortunately you didn't show two lines. You only ended up showing one line, and you can't deduce a pattern from that.

:: vague mental association with some process in mathematics or possibly historical linguistics where you do in fact make a rule based on only one instance ::

mod_rewrite alone can't change the case of an URL. It can detect requests in a case-insensitive way, like picking up .jpg, .Jpg and .JPG in a single rule. But there's no straightforward* way to make them all output .jpg other than by individual redirects. The same applies to any mass replacement like lowline-to-hyphen (or, ahem, vice versa**).

That's why g1 originally said "php script". (If you read the apache subforum regularly, you will see that g1 very often says "php script". Probably because people only come here when they've got questions that would defeat the ordinary htaccess. Nothing to do with the fact that any solution involving php bumps the follow-up questions into a different subforum ;)) Among other things, php can change cases standing on its head. Heck, even javascript can do it. But mod_rewrite can't.


* There's a post by jdmorgan illustrating the non-straightforward way. It relies on flags like [C] and [N] that normal people approach with EXTREME CAUTION.

** Yes, I know, it's just me. But I always think hyphens look spammy. get-your-cheap-widgets-here dot biz, kind of thing.

[Hoople]

This sounds like a classic case of trying to fix the effect and not the cause(s).

How long a history of change as well as how many big issues you are trying to hit with one brushstroke?

Are there multiple site redesign URL changes that are you covering and their time period?

[londrum]

have you just got one htaccess file in the root?

maybe you could split the file up into lots of smaller files, and put each one into the directory that it affects.

that way you wont have to read the entire thing every single time.

[acimag]

I get it. . Thanks Lucy. I see my problem.

Hoop. Its just a massive site we are reduing and the URL's and wanted not to lose search ranking.

Londrum. I think your onto something. Thanks!