301, but not redirecting

Some 301s are redirecting. Others aren't. Why?

[hepcat]

Hi,

Here some examples of rules I have in my httpd.conf file. When I test it, it works as expected. I end up with two entries in my access file. One with the old URL and the 301 error and the next with the new re-written URL and a successful 200.

  <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteLog "/wherever/rewrite_log"
    RewriteLogLevel 1
    RewriteCond %{REQUEST_METHOD} ^TRACE
    RewriteRule .* - [F]
    RewriteCond %{QUERY_STRING} ^(.*)whatever [OR]
    RewriteCond %{QUERY_STRING} ^(.*)uh
    RewriteRule ^.*$ http://wherever.com/cgi-bin/script.cgi? [L,R=301]
  </IfModule>

but it only ever works when I test it. I have dozens of log entries with the 301 message and the rewrite log says it was redirected to my script, but there's no successful run of the script logged in the access log. Plus, my script appends to a file. The file doesn't have anything in it, thus it's not getting called. There were about 2 dozen 301 entries in my access log this weekend and a corresponding number in my rewrite log, but the script it should be redirecting to is not running (i.e. it's not getting called).

I've tried simulating the request logged in the access file, but I get a 408 (timed out) error with a corresponding 408 in the access log. Anyone have any idea what's going on here?

Thanks,
Rob

[jdMorgan]

Without further information about *what* requests timed out, I'd say that the "visitor" who requested the page with either of the two query strings was not a browser or a legitimate search engine robot.

It is entirely up to the client to "follow" a 301 redirect and re-request the resource from the new URL given in the 301 response. So, your assumption that a 301 necessarily will result in a fetch of your new URL is not valid.

In fact, I'm not sure you really want a 301 redirect here, instead of an internal rewrite. I'm not in the habit of 'exposing' the fact that I use scripts to clients, and that's what a 301 will do.

If your code works when you test it, then I'd say the problem is not on your server -- there's something else going on.

One more thought: I assume you've already set the cache-control header on script.cgi to make it non-cacheable. Otherwise, the client may not issue a fetch to your server if it already has a copy in its cache.

Jim

[hepcat]

Thanks. That's good to know. Is there a way to return content based on the sturcture of the URL if the file doesn't exist?

Rob

[jdMorgan]

If I understand you question, then the answer is "Yes, of course. Half the sites on the Web use scripts to generate pages, and none of those pages can be said to exist as files documents on the Web server -- They don't exist until the script creates them and sends them to the client."

Try some searches here for discussions of the difference between external redirects and internal server rewrites. I believe you are looking for a solution involving internally rewriting URLs to a script that will then generate content based on the requested URL. This is how the majority of dynamic-content sites work.

If you are looking specifically for the 'test' used to see if a file exists, then that is usially accomplished with:

RewriteCond %{REQUEST_FILENAME} !-f

although it's usually a good idea to avoid doing this test until you're sure it's needed; Since it invokes a search of your filesystem, it's not particularly efficient. You might want to limit its scope by doing the test only if the requested URI looks like a filename and not a directory, and if it meets any other criteria you can think of to limit the scope. An example would be:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^products/([^.]+)\.htm$ /my_products.php?prod=$1 [L]

Here, the RewriteCond will be processed *only* if the requested URL ends with ".htm" and appears to reside in the "products" directory (this directory may or may not actually exist).

Jim