Using mod rewrite to block traffic from a particular domain referral?

[georgec]

Hi:
I need some help blocking all traffic from a particular domain referral, lets say xxx.com. I know this requires mod rewrite to first detect the referral itself and act accordingly, but don't know where to go from there. Can anyone help?

The problem I'm facing is that starting recently, a sex related domain has been showing up big time in our referral logs. The referral URL itself is some strange cgi redirect script I can't make heads or tails from (redirects to a 404 page), and it's obvious they're not sending actual visitors to my site. I suspect they may simply be hot linking or accessing some non html files on our server, jacking up the referral logs in the process. Using mod rewrite would seem to be the only solution, as something like IP tables blocks the domain itself, but not traffic redirected from and not necessarily originating from the offending domain (referrals).

Thanks much.

[georgec]

Never mind, figured it out. For those who are interested:

RewriteEngine on
Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} xxx\.com
RewriteRule .* - [F,L]

[georgec]

Just a question on syntax- when is it necessary to backslash in a .htaccess file? For example, if the domain I wish to ban above is:

something.xxx.com

Do I need to specifiy it as:

something\.xxx\.com?

I couldn't confirm that it even makes a difference one way or the other.

[jdMorgan]

georgec,

In a mod_rewrite/regular expressions context, an unescaped period means "any single character," so yes, you should precede periods in domain names, IP addresses, and file extensions with a backslash.

Ref: Regular expressions tutorial [etext.lib.virginia.edu]

Also, you don't need an "L" in your [F,L] flag - "L" is built into "F".

Ref: Apache mod_rewrite [httpd.apache.org]

Jim

[berli]

What is the Options +FollowSymlinks part for?

Can it be safely omitted?

[jdMorgan]

It can be safely omitted, unless you get a 500-server error when you omit it. :)

It is needed if the user acccount was not configured with FollowSymLinks in its <directory> section in httpd.conf (server configuration).

Jim

[georgec]

Thanks jdMorgan for the explanation.

[georgec]

Another follow up question if you don't mind. I need to add another rogue referrer to my .htaccess file. Here's naturally what I gather I should do:

RewriteEngine on
Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} xxx\.com
RewriteCond %{HTTP_REFERER} anotherdomain\.com
RewriteRule .* - [F]

Are there any "flags" I need to add to the end of the two lines containing the domains, like [NC] or [OR]? I've seen them in documentations, but can't figure out when they're needed, such as in this case.

Thanks for the help.

[Wizcrafts]

Georgec

Yes, you do need to ad [OR} after the first line, allowing a space after the expression, ie:

RewriteCond %{HTTP_REFERER} xxx\.com [OR]
RewriteCond %{HTTP_REFERER} anotherdomain\.com 
RewriteRule .* - [F]

Also, if there is a possibility that the referer will have some uppercase letters in it, add NC to the switch, as in [NC,OR]

I hope this helps
Wiz

[georgec]

Great Wiz, thanks. I figured I probably need to add [OR] when it comes to blocking (and not accepting) a list of domains.

Thx.

[tschild]

Approaching your problem from another direction: One possible reason for strange referrer entries of this kind is that they want their URL to show up in a publicly viewable statistics page. Are your web server statistics password-protected or are they viewable to the public?

[claus]

allow me to add that it's normally more accurate to ban an IP address than a host name/referrer, as the latter can more easily be manipulated.

/claus

[georgec]

Tschild, our server doesn't generate web stats, just log files I download to then process offline.

claus, thanks for the note. I've considered banning by their IP instead, though in this case it would seem referer is more appropriate. As mentioned, it's some strange sex site.

[claus]

ohyes, it's easier too se what it means. IPs are just numbers. But, consider for a moment how easy it is to go from, say, www.widgetsex.com to xxx.widgetsex.com on the same IP - it'doesn't take hours to make that change ;)
/claus

[arielmeadow]

I tried using this code in my htaccess file, and I seem to have hit an error.

<Limit GET POST>
Order Allow,Deny
Allow from all
</Limit>

rewriteEngine on
Options +FollowSymlinks
rewriteCond %{HTTP_REFERER} offendingsite\.com
rewriteRule .* - [F]

ErrorDocument 403 [mysite.com...]

When I test by clicking the link on the offending site, I get this error message:

"Redirection limit for this URL exceeded. Unable to load the requested page. This may be caused by cookies that are blocked."

I'm thinking it doesn't have anything to do with cookies, and is instead related to my coding. Oops. Any advice?