referrer spam block htaccess doesn't work

referrer spam block

[andy0403]

Hi, I'm using htaccess to block the extensive referrer spam but they still appear within my log file, I might missunderstand something.

this is a short version of my htaccess:

RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-.)?adult(-.).*$ [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-.)?backside(-.).*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-.)?words(-.).*$ [NC,OR]
# without domain extension
RewriteCond %{HTTP_REFERER} ^http://(www\.)?trafbanf2.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?trafbang.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?threethrees.*$ [NC,OR]
# and now with domain extension
RewriteCond %{HTTP_REFERER} ^http://(www\.)?seeontv.com*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?self-improving.com*$ [NC,OR]
RewriteRule .* - [F,L]

After I put it online they still show up in my awstats and in my raw logfiles with 200 response, schouldn't they get a 403? If I block the IP address they get a 403 but can't use IP blocking because they are not all from static IP addresses.
Any help would be highly appreciated, thank you so much, Andy

[edited by: jdMorgan at 10:53 pm (utc) on Jan. 4, 2006]
[edit reason] Obscure referrers and clean up keywords. [/edit]

[jdMorgan]

andy0403,

Welcome to WebmasterWorld!

You must not include an [OR] flag on your last RewriteCond. These are used to logically OR RewriteConds, and have no meaning if you try to OR a RewriteCond with a RewriteRule. Your rule will be ignored, and you should see a syntax error warning in your server error log.

You can also simplify and shorten the code by removing all instances of ".*$" from your RewriteCond patterns. These characters at the end of a pattern do not change the bahaviour and just take up processing time.

Also, [L] used with [F] is redundant and can be eliminated.

Jim

[andy0403]

thank you very much Jim, you are very helpful as always

The one below would be a go then:
RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-.)?adult(-.) [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-.)?backside(-.) [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-.)?words(-.) [NC,OR]
# without domain extension
RewriteCond %{HTTP_REFERER} ^http://(www\.)?trafbanf2 [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?trafbang [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?threethrees [NC,OR]
# and now with domain extension
RewriteCond %{HTTP_REFERER} ^http://(www\.)?seeontv.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?self-improving.com [NC]
RewriteRule .* - [F]

thanks again, Andy

[jdMorgan]

Not sure... You've got "(-.)" in several places, and I'm not sure what you intend by it.

The meaning is "match a hyphen followed by any character, and create a back-reference containing those two characters for later use. I'm sure you don't need the back-reference, so that reduces the required pattern to "-." and saves some CPU time. I assume you're trying to match <domain name> hyphen <any character> with it...

Jim

[andy0403]

thank you again Jim.
Honest the first 3 lines I got from another fight referrer spam site example and I really don't know what they try to do, they are actually not my biggest concern, porn spam is very rare on my domain, it's more the "stealth advertising" software refferer spam that concerns me, it is getting out of control I guess because my site has pretty good listings in search engines and they screw up my log files completely.

Now I did what you suggested but still no success, I still have them in my log files:
ip68-109-***-131.lu.dl.cox.net - - [04/Jan/2006:21:53:50 -0800] "GET / HTTP/1.1" 200 28838 "http://changedbyme.com" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"
ip68-109-***-131.lu.dl.cox.net - - [04/Jan/2006:21:53:50 -0800] "GET / HTTP/1.1" 200 28832 ".) http: //www. changedbyme.com/vsearch.php" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"
ip68-109-***-131.lu.dl.cox.net - - [04/Jan/2006:21:53:50 -0800] "GET / HTTP/1.1" 200 28830 "http: //www. changedbyme.com/voodoo.php" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"

even I have this line in my htaccess for this referrer:
RewriteEngine on
#Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} ^http://(www\.)?changedbyme.com [NC,OR]
RewriteRule .* - [F]

They still get a 200 server response but it should be a 403. I know I could do it over the ip address but some of them change their ip address from time to time.
BTW I changed the domain name of the spam site and "destroyed" the url... just in case.
Do you have any suggestions? Thanks again, Andy

[edited by: jdMorgan at 5:17 pm (utc) on Jan. 5, 2006]
[edit reason] Obscured IP address in hostname [/edit]

[jdMorgan]

1) Are you sure you don't need to uncomment the "Options" directive?
2) What (if anything) is in your server error log for these requests?
3) If the request is directed to an 'aliased' directory, such as a script outside your HTTP-accessible directory structure, then your .htaccess file won't have any effect. The code has to be in the directory-path of the files to be protected.

The code should work fine if mod_rewrite is available to you, is enabled by the FollowSymLinks option, and if the code is in a location where it will be executed for the requests.

Note that these attempts will still appear in your logs; The only change will be that the server response code will show as a 403-Forbidden, and the byte count will reflect that of your 403 error page.

Jim

[andy0403]

thanks Jim,

1. I will try both ways and see what happens
2. I can't see entries made by these referrers in the error logs
3. the htaccess is in the same folder as the html files, so it should work fine, all other things setup in htaccess work just fine as redirect for example.

mod_rewrite is available

I know they still will show up in my logs but they all have a 200 and not 403 unless I block the IP they used, then it will be 403, but I need to get it working with the referrer, it's really bad.

Andy

[andy0403]

A little update:
I did uncomment

Options +FollowSymlinks
no changes! I still get unbelievable hits from them and just today additional 16 new referrer spam domains from noon til now. They all get a 200 server response.
I will put this comment back and see what happens.
Andy

[jdMorgan]

I suspect that the problem is caused by something I can't see on this forum. However, the code that I can see looks correct. All I can recommend is this:

Make sure the final referrer RewriteCond does NOT have an [OR] flag on it.

Make sure that *every* referrer RewriteCond above it has DOES have an [OR] flag on it.

Make sure that every "{" character is actually a "{" and not a "(".

Same for the "}" character

If any of these is not true, then the rule will fail.

This stuff (mod_rewrite) requires absolute precision, but there is no "magic" to it. Something is causing this code to fail or perhaps it is not running at all due to a problem outside this code itself.

If you cannot find any problems like those listed above, then start with a very short list of RewriteConds for referrers (like the three worst ones), and test that. If it works add just a few more and test again.

Jim

[andy0403]

thank you Jim,
I'll follow your recommendations.
BTW the last condidtion never contained an OR, I made the mistake just here in the forum, the original file was fine.
Andy

[andy0403]

Hi Jim,
finally I got it to work

RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} ^http://(www\.)blabla\.net [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)blabla-report-report\.co\.uk [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)blacky-mbl\.de [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)xxblablahomes [NC]
RewriteRule .* - [F]

These rules work with e.g.
[xxblablahomes.com...]
[xxblablahomes.net...]
[xxblablahomes.info...]

but they still fail with
xxblablahomes.com
xxblablahomes.net
xxblablahomes.info

I checked it in wannabrowser and also from a friends site I did add for test reasons.
Can you pls point me to the right direction to get it also work without the http and www, I'm sort of stuck with that, I'm so tired.... ;-) lol
Thank you again Jim, you're really a great help, Andy

[jdMorgan]

It is unusual to get referrers without the "http://", so I'm not sure I'm following along with the problem. However, the referer may or may not contain the www subdomain -- That's up to the webmasters of the other sites.

To make the www optional, simply add a "?" after the paranthesized www -- that's why they were parenthesized in the first place:

RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} ^http://(www\.)?blabla\.net [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?blabla-report-report\.co\.uk [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?blacky-mbl\.de [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?xxblablahomes [NC]
RewriteRule .* - [F]

The "?" in a regex pattern makes the preceding character or parenthesized group of characters optional.

For more information on regular expressions, see the regex tutorial cited in our forum charter [webmasterworld.com]

Jim

[andy0403]

Hi Jim,
I got it running pretty good now, it catches all of them, there is just one smart a** who changes their referer every single time, well some of it I did catch but there is one thing remaining and I have a problem with. He also put signs in front of the http such as

+http://blabla.de

and then he slip through.
Can you please give me a hint how can I catch any sign in front of the http? Is there a way at all?
Andy

[jdMorgan]

Add an optional "+" sign ahead of http:

RewriteCond %{HTTP_REFERER} ^\+?http://(www\.)?blabla\.net [OR]

or just remove the start anchor from the pattern.

RewriteCond %{HTTP_REFERER} http://(www\.)?blabla\.net [OR]

The first method is less likely to cause unexpected results, for example with cgi-type referrers.

Jim

[andy0403]

works like a charm Jim, I appreciate your help!