Replacement URL query string

Replacement URL query string - & is replaced

[batface]

Oh dear, I have a problem with a rewrite that is confusing me.

http://www.example.com/cgi-bin/lbitem-dba.cgi/C23-4170

to

http://www.example.com/tools/?search=1&keywords=C23-4170&product-list-search-commerce-products=Search

So the rule I have is -

Rewrite ^/cgi-bin/lbitem-dba\.cgi/C23-4170$ http://www.example.com/tools/?search=1&keywords=C23-4170&product-list-search-commerce-products=Search [R=301,L]

The output is this -

http://www.example.com/tools/?search=1/cgi-bin/lbitem-dba.cgi/C23-4170keywords=C23-4170/cgi-bin/lbitem-dba.cgi/C23-4170product-list-search-commerce-products=Search

The original URL is being inserted with help of the '&' placeholder?

What am I doing wrong? Please keep this simple for my childlike brain! :)

edited- error in my text

[lucy24]

Are you positive your cat didn't walk across the keyboard, changing the rule to

RewriteRule ^/(cgi-bin/lbitem-dba\.cgi/C23-4170)$ http://www.example.com/tools/?search=$1&keywords=C23-4170&product-list-search-commerce-products=Search [R=301,L]

?

I concede that this may not literally be what happened-- maybe you've got a ferret, not a cat-- but we need to start with the simplest explanation.

Is this rule lying loose in your config file? Is there any extra stuff in place pertaining to handling of the .cgi extension?

<topic drift>
Are you really redirecting (externally) from a short pretty URL to a long ugly one? Why?
</td>

Edit: That would be the same cat, or perhaps ferret, that changed the intended "RewriteRule" to "Rewrite" alone. (I assume the full "RewriteRule" is present in the real thing, or you'd just get an error.)

[batface]

Hi Lucy24,

yes I do have 2 cats. I concede that I once had v space h kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk(14 thousand of them) on a Word doc after leaving them in charge while I popped to the shop one day, but on this occasion they were far too tired to wind me up. The purring may have put me off finishing my Rewrite word though (on here)!

<drift reply>The redirecting URL is not up to me just following orders.</drift reply>

Ok, I looked right through the file, lots of stuff in there and there were a couple of lines possibly messing things up. I have cleaned those lines out, tested and site functions as it should.

I've also tested the rule to RedirectMatch and with both rules the output URL is the same with the results posted above.

The rule is definitely not ... ^/(cgi-bin/lbitem-dba\.cgi/C23-4170)$
It was either ^/cgi-bin/lbitem-dba\.cgi/C23-4170$ to test a single defined URL or last test

RedirectMatch 301 ^/cgi-bin/lbitem-dba\.cgi/(.*)$ http://www.example.com/tools/?search=1&keywords=$1&product-list-search-commerce-products=Search

[not2easy]

If you already have any RewriteRules in your htaccess file, you should not use RedirectMatch. You may have unwanted results. I would check the live headers to see what the rewrite is doing.

[batface]

It's a straight 301 redirect and 200 ok for the new 'weird' URL.

[lucy24]

Yes, don't mix mod_alias ("Redirect" by that name) and mod_rewrite for redirects. The problem is that mod_rewrite executes first, so you'll have things happening in the wrong order, with the added possible risk of double redirects. If you have any existing rules using Redirect(Match), change them to use mod_rewrite and then put them all in optimal order.

there were a couple of lines possibly messing things up. I have cleaned those lines out, tested and site functions as it should.

Been there. Done that.

[whitespace]

Rewrite ^/cgi-bin/lbitem-dba\.cgi/C23-4170$ http://www.example.com/tools/?search=1&keywords=C23-4170&product-list-search-commerce-products=Search [R=301,L]

Lucy24 has already queried this, but I don't see a response.... (?)

1. "Rewrite" should be "RewriteRule" - I don't how this could be a forum typo (I assume it's a copy and paste) - so this would seem to be an error? But maybe not? We're not on Nginx are we?

2. Where is this directive? In .htaccess / Directory section or server config. The above pattern would not match in .htaccess.

[batface]

1. No, this is on the forum. I may have pasted but I also titted around with it. No not Nginx. The htaccess is functioning and there are a couple of RewriteRule rules in there that work fine.

2. The .htaccess file is in the root. There is also some other Redirect 301/cgi-bin/lbcat-dba.cgi/SomeFile ... in there that functions. Why would the above not work in .htaccess?

[batface]

Is this rule lying loose in your config file? Is there any extra stuff in place pertaining to handling of the .cgi extension?

Gone through everything and there does not appear to be anything in the config file or anything else relating to .cgi ?

[whitespace]

Why would the above not work in .htaccess?

Because the RewriteRule pattern starts with a slash. In .htaccess files, the directory-prefix is first removed from the URL-path that matches the pattern (this contrasts with the server config). So the pattern should start ^cgi-bin not ^/cgi-bin

[batface]

Ok, here is where i'm at this very moment

RedirectMatch as above with the extra added stuff.

The page being redirected does not exist so it returns a 404 header response.
RewriteRule as per instructions returns a 404 header so the rule is not recognising the URL to rewrite it.
RewriteRule ^cgi-bin/lbitem-dba\.cgi/C23-4170$ http://www.example.com/tools/?search=1&keywords=C23-4170&product-list-search-commerce-products=Search [R=301,L]

I'm at a loss?

[batface]

This is taken from http.conf

ServerName example.com
ServerAlias www.example.com
DocumentRoot /home/exampler/public_html
ServerAdmin [e-mail]
UseCanonicalName Off
Options -ExecCGI -Includes
RemoveHandler cgi-script .cgi .pl .plx .ppl .perl
CustomLog /usr/local/apache/domlogs/example.com combined
CustomLog /usr/local/apache/domlogs/example.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
## User exampler # Needed for Cpanel::ApacheConf
UserDir enabled exampler

suPHP_UserGroup exampler exampler



SuexecUserGroup exampler exampler



RMode config
RUidGid exampler exampler

# To customize this VirtualHost use an include file at the following location
# Include "/usr/local/apache/conf/userdata/std/2_2/exampler/example.com/*.conf"



ServerName example.com
ServerAlias www.example.com
DocumentRoot /home/exampler/public_html
ServerAdmin [e-mail]
UseCanonicalName Off
Options -ExecCGI -Includes
RemoveHandler cgi-script .cgi .pl .plx .ppl .perl
CustomLog /usr/local/apache/domlogs/example.com combined
CustomLog /usr/local/apache/domlogs/example.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
## User exampler # Needed for Cpanel::ApacheConf
UserDir enabled exampler

I read there is typically 'AllowOverride none' directive on the cgi-bin folder/alias, so .htaccess files in the folder/alias will be ignored. In which case update to 'AllowOverride Fileinfo'. Although 'AllowOverride none' is not present am I getting warm (if I want the RewriteRule to function)?

[lucy24]

Although 'AllowOverride none' is not present am I getting warm (if I want the RewriteRule to function)?

Does the AllowOverride statement occur anywhere at all in the config file?

:: detour to docs to refresh memory ::

Oh, yikes, glad I looked. Up through 2.3.8 (meaning, in practice, Apache 2.2 and earlier) the default is All. From 2.3.9 on (i.e. 2.4) the default flipflops to None. (This is probably a good idea, since it means htaccess can't be used unless you expicitly say so.) What version are you on?

It is always a good idea to test some random rule just to verify that the module-- mod_rewrite or whatever it may be-- is working at all. Say something like

RewriteRule widget http://example.com/foobar [R=301,L]

(using any two nonsense words that don't actually occur in URLs). Then request anything with "widget" in the URL, and see if you end up on your 404 page with the address bar saying "foobar" or whatever it is.

[batface]

No that rule does not occur. I sneaked in a <?php phpinfo() ?> against their rules and discovered the server is Litespeed.

However, rewriteRule does function as you suggest. I tested with your example, I also tested with a live page - both return the correct URL (the former 404, the latter 200).

Within the htaccess there is a rule - Redirect 301/cgi-bin/lbcat-dba.cgi/PJustBefore http://www.example.com/directory/directory/
which was reported to me as working just fine.

I commented out and tried that also with a RewriteRule and again get a 404 page at example.com/cgi-bin/lbcat-dba.cgi/PJustBefore

I reverted to the Redirect 301 and now also get a 404 page at example.com/cgi-bin/lbcat-dba.cgi/PJustBefore, so assume there must be some internal cache thing going on (yes I tested on a clean browser).

The 404 page returned in the above examples (apart from your test) is the server 404 not the CMS based 404. I didn't pick up the very subtle "Proudly powered by LiteSpeed Web Server" plastered at the bottom of the page. I've wiped my glasses! But nevertheless RewriteRule functions away from that cgi-bin directory.

aaahhhhh! What to do?

[batface]

...also the content in /cgi-bin/ is dynamically generated, no files/folders. Not sure why RedirectMatch functions (albeit incorrectly) and RewriteRule doesn't.

[lucy24]

There's a CMS involved you say? Then there is almost certainly a rule somewhere saying !-f ("If the requested file doesn't physically exist.") Any RewriteRules involving external redirects have to go before the CMS boilerplate. But really, you've got to get rid of the mod_alias rules, or things will be a big mess.

Does the directory itself -- the /cgi-bin/ part -- physically exist? If so, what does it contain? Just an "index.cgi" file or similar?

:: vaguely looking around for whitespace, as I've got my annual cold and will not be able to say anything intelligent for the next week or so ::

[batface]

Yes there is:

# -- concrete5 urls start --
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L]
</IfModule>
# -- concrete5 urls end --

This is at the foot of the htaccess.

The /cgi-bin/ directory exists and is empty.

The RewriteRule created is above any of the other Redirect 301s.

[batface]

...I can get those Redirect 301s concerted to RewriteRule.

Edit: Done

[lucy24]

The RewriteRule created is above any of the other Redirect 301s.

Doesn't matter. Each module is an island. You could say
RewriteRule blahblah
RedirectMatch blahblah
Deny from blahblah
SetEnvIf blahblah
RewriteRule blahblah
Allow from blahblah
SetEnvIf blahblah
RedirectMatch blahblah
and the server would still go through, picking out each module's directives in a predetermined order while ignoring all others. The only thing that makes a difference is ordering of rules within a given module. Sometimes not even then: The "Order" directive determines who gets read first, Allow or Deny. That's assuming 2.2 or earlier. Did you ever say what version you're on?

I can get those Redirect 301s concerted to RewriteRule.

Yup. Any redirect done with mod_alias can be converted to mod_rewrite. (The opposite isn't true.)

In general, with exceptions, all your RewriteRules should be listed in order of severity:
-- access control first (no point in redirecting someone who's going to be locked out by the same module)
-- [G] flag if any
-- external redirects ([R] flag, with or without 301)
-- internal rewrites ([L] flag alone
-- rarely, no-flag rules such as setting cookies
And then, within each category, go from specific to general.

One big exception is files that you need everyone to be able to see, so lines like
RewriteRule ^my403\.html - [L]
go before everything else.

index.php/$1 [L]

Do please put a / at the front of that target. If you do this consistently, you don't even need the RewriteBase directive, and it's safer. Is it really / (directory slash) and not ? (query string)?

Major question
Is the stuff in the /cgi-bin/ directory connected with the CMS, or is it an entirely separate process? If it's unrelated, you'll need a supplementary RewriteCond saying something like

RewriteRule %{REQUEST_URI} !^/cgi-bin/

(There are other ways to achive the same result. The point is to ensure that the CMS doesn't see requests that it wasn't supposed to handle.)