Mod rewrite errors since apache 2.2.10 update

[Dreamcontest]

I have a website that is no longer functioning due to my hosting companies upgrade to Apache web server to version 2.2.10 (from 1.3.4x). The problem is in the .htaccess file.

This is the file;

# Version: 1.02 (03.07.2004.)
# Options +FollowSymLinks
RewriteEngine on
#RewriteBase /
RewriteRule  ^admin/$              admin/index.php [L]
RewriteRule  ^/$                index.php?req=search [L]
RewriteRule  ^paid/(.*)$            index.php?req=paid&id=$1 [L]
RewriteRule  ^scripts/(.*)/(.*).html$      index.php?req=cat&id=$1&offs=$2 [L]
RewriteRule  ^scripts/(.*)$           index.php?req=cat&id=$1 [L]
RewriteRule  ^add/(.*)$             index.php?req=add&id=$1 [L]
RewriteRule  ^confirm/([^/]+)/(.*)$       index.php?req=confirm&memb_id=$1&code=$2 [L]
RewriteRule  ^detailed/(.*).html$        index.php?req=detailed&id=$1 [L]
RewriteRule  ^pages/(.*).html$         index.php?req=pages&id=$1 [L]
RewriteRule  ^addreview/(.*)$          index.php?req=addreview&id=$1 [L]
RewriteRule  ^reviews/(.*)$           index.php?req=reviews&id=$1 [L]
RewriteRule  ^subscribe/(.*)$          index.php?req=subscribe&id=$1 [L]
RewriteRule  ^news/(.*)$            index.php?req=news_read&id=$1 [L]
RewriteRule  ^register\.html$          index.php?req=register [L]
RewriteRule  ^login\.html$           index.php?req=login [L]
RewriteRule  ^logout\.html$           index.php?req=logout [L]
RewriteRule  ^members\.html$          index.php?req=members [L]
RewriteRule  ^modify\.html$           index.php?req=modify [L]
RewriteRule  ^modify/(.*)$           index.php?req=modify&id=$1 [L]
RewriteRule  ^refer/(.*)$            index.php?req=refer&id=$1 [L]
RewriteRule  ^report/(.*)$           index.php?req=report&id=$1 [L]
RewriteRule  ^popular\.html$          index.php?req=popular [L]
RewriteRule  ^top\.html$            index.php?req=top [L]
RewriteRule  ^new\.html$            index.php?req=new [L]
RewriteRule  ^new/([^/]+)/(.*)/(.*).html$    index.php?req=new&id=$1&day=$2&offs=$3 [L]
RewriteRule  ^new/([^/]+)/(.*)$         index.php?req=new&id=$1&day=$2 [L]
RewriteRule  ^addrelated\.html$         index.php?req=addrelated [L]
RewriteRule  ^resendcode\.html$         index.php?req=resendcode [L]
RewriteRule  ^emailpass\.html$         index.php?req=emailpass [L]
RewriteRule  ^usersettings\.html$        index.php?req=usersettings [L]
RewriteRule  ^search\.html$           index.php?req=advancedsearch [L]
RewriteRule  ^link\.html$            index.php?req=link_to_us [L]
RewriteRule  ^aboutus\.html$          index.php?req=aboutus [L]
RewriteRule  ^sitemap\.html$          index.php?req=sitemap [L]
RewriteRule  ^advertise\.html$         index.php?req=advertise [L]
RewriteRule  ^rss/(.*)$             rss.php?$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)/(.*).html$          index.php?req=cat&id=$1&offs=$2 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)/index\.html$         index.php?req=cat&id=$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)/$               index.php?req=cat&id=$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)\.html$            index.php?req=detailed&id=$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)$               index.php?req=cat&id=$1 [L]
<Files ~ "License Number.txt">
  Order allow,deny
  Deny from All
</Files>
# set the server timezone
SetEnv TZ America/New_York
AddCharset iso-8859-1 .html
AddCharset iso-8859-1 .php
<Files 403.shtml>
order allow,deny
allow from all
</Files>

Since the upgrade to the new version of Apache the website hardly loads, images dont show and things crawl along.

I have located that this specific part causes the issues.

RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)/(.*).html$          index.php?req=cat&id=$1&offs=$2 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)/index\.html$         index.php?req=cat&id=$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)/$               index.php?req=cat&id=$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)\.html$            index.php?req=detailed&id=$1 [L]
RewriteCond %{REQUEST_FILENAME} !-F
RewriteCond %{REQUEST_FILENAME} !-D
RewriteRule ^(.*)$               index.php?req=cat&id=$1 [L]

Contacting the host has given me this - "We've placed the rewrite rules in place and we are seeing that they do work but we're seeing the same slowness that you're indicating. Unfortunately due to the complexity of regex in mod_rewrite rules we support mod_rewrite on the servers but the actual regex and rules themselves is something that we can't directly support and is outside the scope of our normal support so this is something you'll need to investigate and address due to this."

Isnt that nice? Hope someone here can help me out as the site is not functioning :(

[jdMorgan]

I suggest that you pay particular attention to the differences between -F and -f, and -D and -d, as documented (and specifically warned-about) in the mod_rewrite RewriteCond documentation [httpd.apache.org]...

Also, in all instances where you've used ".*" in a pattern, followed by anything else, look at using a "stop matching on next 'x' character" construct, such as this, which stops matching on the first period that it finds:

^([^.]+)\.html$

Changing from ".*" to such a more-specific pattern will also speed up your server quite a bit, especially when multiple ".*" subpatterns are used. Using more-specific subpatterns can make a rule such as

RewriteRule ^new/(.*)/(.*)/(.*).html$ index.php?req=new&id=$1&day=$2&offs=$3 [L] 

run hundreds of times faster if you use

RewriteRule ^new/([^/]+)/([^/]+)/([^.]+)\.html$ index.php?req=new&id=$1&day=$2&offs=$3 [L] 

instead.

Jim

[edited by: jdMorgan at 12:46 am (utc) on Nov. 18, 2008]

[jdMorgan]

Also, I'd suggest replacing your very last rule above with

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^.]+)$ index.php?req=cat&id=$1 [L]

to avoid checking "file exists" and "directory exists" for each and every image or URL-path request which includes a filetype (rejected because of the "not a period" in the pattern). These filesystem checks should be avoided at all costs, because they are hugely expensive in CPU and disk-access time. Since RewriteConds are not processed unless the RewriteRule pattern matches, you can avoid unnecessary filesystem checks by using very-specific RewriteRule patterns.

Jim

[g1smd]

Yow. What a nightmare, for a four year old configuration to suddenly fall over after an upgrade.

This stuff is very unforgiving of deviation from the documentation; I spent literally hours with an Error 500 message last week due to a simple typo that had actually worked on Apache 1.3.7 and then failed horribly on Apache 2.2.3.

Hmm. Apache 2.2.10. Must bug the hosts about upgrading. Looks like they might be getting behind.

[Dreamcontest]

I appreciate the response... i just wanted to let all who else who may have this problem that changing to

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^.]+)$ index.php?req=cat&id=$1 [L]

worked....

i looked into changing some of the other things above as well.. thank you

[jdMorgan]

I would make it a high priority to replace all occurrences of the "-F" RewriteCond token with "-f" and to replace all "-D" RewriteCond tokens with "-d". That, along with the change you already made as noted above, plus replacement of the ".*" patterns, should make a very dramatic improvement in your server's performance.

Unless there is something very non-obvious about your site, there should be no difference whatsoever in rewriting behavior, but a very big gain in speed.

In case it is not obvious:

Replace each occurrence of "(.*)/" with "([^/]+)/"
Replace each occurrence of "(.*)." or the more-correct "(.*)\." with "([^.]+)\."

For the "trailing" "(.*)$" patterns in your rules, I'd suggest trying the replacement "([^/]+)$" if you want to require the corresponding query parameter to be non-blank, or "([^/]*)$" if you do want to allow it to be blank.

In both cases, the pattern will reject a trailing URL-path-part if it contains a slash. Based on your URL-to-query-string mapping scheme of one slash-delimited parameter in the URL mapping to one query-string name/value pair, this would prevent a match if there was an additional parameter in the requested URL, meaning that you probably don't want to invoke that rule because it would then put two parameters into one name/value pair.

If you really do want to pass blank parameters *and* parameters containing slashes into the query string, then stick with "(.*)$".

Jim