mod_rewrite: UA redirect looping, How to exclude target page?

mod_rewrite: UA redirect to page looping, How to exclude target page?

[MattyUK]

Hi

I have a useragent redirect that is loopoing and could use a hand to see if it is possible.

I want to direct specific user agents to a specific page. I want to exclude that page from checking the user-agent so that the rule doesn't loop.

I got this so far.

RewriteCond %{HTTP_USER_AGENT} pmafind [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Fetch\ API\ Request [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DFind [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cfetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} POE-Component [NC,OR]
RewriteRule .* http://www.example.co.uk/no_hack.html [NC,L]

But they obviously loops since the no_hack.html is processed and matches on the same rule since the user-agent hasn't changed. So time to exclude the target page from the rule.

RewriteCond %{REQUEST_URI}!^/no_hack\.html [NC,AND]
I figured adding this as the top condition would do it. It loops or I get a 500 server error.

So the entire rule is:
RewriteCond %{REQUEST_URI}!^/no_hack\.html [NC,AND]
RewriteCond %{HTTP_USER_AGENT} pmafind [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Fetch\ API\ Request [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DFind [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cfetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} POE-Component [NC,OR]
RewriteRule .* http://www.example.co.uk/no_hack.html [NC,L]

Can anyone tell me where I am going wrong please?

MattyUK

[edited by: jdMorgan at 3:01 pm (utc) on May 17, 2006]
[edit reason] Examplified. [/edit]

[jdMorgan]

Minor syntax problems:

1) Space required between "}" and "!"
2) There's no such flag as "AND" because "AND" is the default -- see the documentation.
3) End-anchor the excluded URL (not an error, just recommended)
4) Last RewriteCond must NOT have an [OR] on it.

RewriteCond %{REQUEST_URI} !^/no_hack\.html$ [NC]
RewriteCond %{HTTP_USER_AGENT} pmafind [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Fetch\ API\ Request [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DFind [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cfetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} POE-Component [b][NC][/b]
RewriteRule .* http://www.example.co.uk/no_hack.html [NC,L]

Shorter equivalent version:

RewriteCond %{REQUEST_URI} !^/no_hack\.html$ [NC]
RewriteCond %{HTTP_USER_AGENT} cfetch¦DFind¦Fetch\ API\ Request¦pmafind¦POE-Component¦wget [NC]
RewriteRule .* http://www.example.co.uk/no_hack.html [NC,L]

None of these UAs are likely to follow the redirect or 'read' your no_hack page. For that reason, you might as well just 403 them:

RewriteCond %{REQUEST_URI} !^/custom_403\.html$ [NC]
RewriteCond %{HTTP_USER_AGENT} cfetch¦DFind¦Fetch\ API\ Request¦pmafind¦POE-Component¦wget [NC]
RewriteRule .* - [F]

The exclusion for the custom error page is only needed if you actually have a custom 403 error page.

As usual, change all broken pipe "¦" characters above to solid pipes before use; Posting on this forum modifies pipe characters.

Jim

[MattyUK]

Thanks jdmorgan

1) Space required between "}" and "!"
Something in the forum posting does this. Try it. I was finding all space infront of! (see it did it then too) was removed on posting. I should have put a comment in however.

2) There's no such flag as "AND" because "AND" is the default -- see the documentation.
Thanks I read up, your right. My bad. I have no idea where I got that daft idea from.

3) End-anchor the excluded URL (not an error, just recommended)
Ok, that makes sense. Ta.

4) Last RewriteCond must NOT have an [OR] on it.
My bad again. I had pasted from when I tried the exclusion as the last condition in the chain.

That page (aside from delivering a largely unread message) has desired code that activates when accessed. Hence their direction there rather than a straight 403.

Thank you for a very helpful post. :D

[jdMorgan]

If you want to activate a script, I'd suggest an internal rewrite rather than a redirect. One characteristic that you can use to identify 'bad' user-agents is that they often do not follow redirects -- check your log file to confirm. Using an internal rewrite removes the client's option to follow or not follow a redirect; The no_hack file is served instead of the requested URI, and the client has no choice in the matter.

The form of the rule changes just a bit for an internal rewrite.

RewriteRule .* /no_hack.html [L]

Note that the [NC] flag is never needed with a pattern of ".*" -- a minor point I missed in the code posted above.

If you do wish to continue to use an external redirect, I'd suggest using the [R=301,L] or [R=302,L] flag on the previously-posted rules to specify a 301 or 302 redirect, just to make the code explicit for later reference.

Jim

[MattyUK]

Wonderful. Helpful again. Thank you very much. I'm learning here.

Your insight caused me to think and I was about to ask:

"Assuming the redirect is always to the same site. When wouldn't you use internal redirects?"

But then I answered it as:
When you want the request URL to be a specific format (i.e. /dir/value1/value2/ rather than /page.php?name1=value1&name2=value2

But then can't you internally redirect to "/dir/value1/value2/" since the whole request is parsed again?
(I just ran a few tests and figured out you CAN redirect internaly to urls that are matched within the .htaccess rather than only to files that exist locally. So the above answer is void and becomes "When you want to redirect to an external site." lol)

I guess I am trying to better understand when to use the 3 methods I know of:
Proxy:
RewriteRule /testMatch/ http://www.example.com/testMatch.php [P,L]

External Redirect:
RewriteRule /testMatch/ http://www.example.com/testMatch.php [L]

Internal Redirect:
RewriteRule /testMatch/ /testMatch.php [L]

(I've deliberatly left out optional the R={code} values on the quick examples above. I didn't know about the R={code} flag before. Thanks)

Do you have any rule of thumb guidance?