Welcome to WebmasterWorld Guest from 54.147.236.192

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

REFERER Overkill?

REFERER, Overkill, block

     
3:24 am on Mar 2, 2014 (gmt 0)

Junior Member

joined:Jan 12, 2014
posts:95
votes: 0


REFERER Overkill?

Working on a new site that Starts w/ .fr the REFERER zaps the CSS in IE only...

I always thought it blocked the domains ending in .fr?

I have also seen some iPads & sometimes rr.com, comcast.net & myvzw.com being blocked.


# Referrer Blocker - .se search bug?
RewriteEngine On
RewriteCond %{HTTP_REFERER} \.at [NC,OR]
RewriteCond %{HTTP_REFERER} \.biz [NC,OR]
RewriteCond %{HTTP_REFERER} \.br [NC,OR]
RewriteCond %{HTTP_REFERER} \.by [NC,OR]
RewriteCond %{HTTP_REFERER} \.cn [NC,OR]
RewriteCond %{HTTP_REFERER} \.cy [NC,OR]
RewriteCond %{HTTP_REFERER} \.cz [NC,OR]
RewriteCond %{HTTP_REFERER} \.de [NC,OR]
RewriteCond %{HTTP_REFERER} \.es [NC,OR]
RewriteCond %{HTTP_REFERER} \.eu [NC,OR]
# RewriteCond %{HTTP_REFERER} \.fr [NC,OR]
RewriteCond %{HTTP_REFERER} \.hk [NC,OR]
RewriteCond %{HTTP_REFERER} \.in [NC,OR]
RewriteCond %{HTTP_REFERER} \.info [NC,OR]
RewriteCond %{HTTP_REFERER} \.ir [NC,OR]
RewriteCond %{HTTP_REFERER} \.it [NC,OR]
RewriteCond %{HTTP_REFERER} \.jp [NC,OR]
RewriteCond %{HTTP_REFERER} \.kr [NC,OR]
RewriteCond %{HTTP_REFERER} \.kw [NC,OR]
RewriteCond %{HTTP_REFERER} \.lt [NC,OR]
RewriteCond %{HTTP_REFERER} \.lu [NC,OR]
RewriteCond %{HTTP_REFERER} \.lv [NC,OR]
RewriteCond %{HTTP_REFERER} \.mx [NC,OR]
RewriteCond %{HTTP_REFERER} \.nl [NC,OR]
RewriteCond %{HTTP_REFERER} \.pk [NC,OR]
RewriteCond %{HTTP_REFERER} \.pl [NC,OR]
RewriteCond %{HTTP_REFERER} \.pw [NC,OR]
RewriteCond %{HTTP_REFERER} \.py [NC,OR]
RewriteCond %{HTTP_REFERER} \.ro [NC,OR]
RewriteCond %{HTTP_REFERER} \.ru [NC,OR]
RewriteCond %{HTTP_REFERER} \.sk [NC,OR]
RewriteCond %{HTTP_REFERER} \.su [NC,OR]
RewriteCond %{HTTP_REFERER} \.to [NC,OR]
RewriteCond %{HTTP_REFERER} \.tr [NC,OR]
RewriteCond %{HTTP_REFERER} \.ua [NC,OR]
RewriteCond %{HTTP_REFERER} \.uy [NC,OR]
RewriteCond %{HTTP_REFERER} \.vg [NC,OR]
RewriteCond %{HTTP_REFERER} \.vn [NC,OR]
RewriteCond %{HTTP_REFERER} \.xxx [NC]
RewriteRule .* - [F]
4:03 am on Mar 2, 2014 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10589
votes: 21


it will block any request that has ".fr" anywhere in the HTTP Referer header url, not just for the .ccTLD
(assuming that RewriteCond isn't commented out)
5:06 am on Mar 2, 2014 (gmt 0)

Junior Member

joined:Jan 12, 2014
posts:95
votes: 0


I know (deny from .fr) works too, but I also want to do it from the bad REFERER like some-forum.fr
8:03 am on Mar 2, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


You need to end-anchor each TLD pattern.

There's a simpler way to do OR operations: \.(xx|yy|zz)$
9:52 am on Mar 2, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


...and
\.(aa|af|aj|az|ba|bb|bc|bd|be|cg|da|db|dc|dd|de|dh|dk|dm|dr|ds|dt|du|dz|ep|...)$

simplifies to
\.(a[afjz]|b[a-e]|cg|d[a-ehkmr-uz]|ep|...)$

and processes massively faster.
12:57 pm on Mar 2, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13256
votes: 361


Working on a new site that Starts w/ .fr the REFERER zaps the CSS in IE only...

I've looked at that upside-down, backward and sideways, and still can't make sense of it.

You need to end-anchor each TLD pattern.

But the referer is the whole page, not just the domain. Wouldn't it have to be ($|/) at the end?

With that many bad referers, seems like it would be easier to whitelist:

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !\.(com|org|net|uk|au)($|/)


... and constrain the rule to requests for pages:
(\.html|/|^)$

substituting whatever extensions you actually use. Non-page requests would have a different set of rules with even tighter restrictions.
1:10 pm on Mar 2, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Err, yes, end the pattern with a slash or indeed whitelist rather than blacklist.
12:19 am on Mar 3, 2014 (gmt 0)

Junior Member

joined:Jan 12, 2014
posts:95
votes: 0


How does this look?

# Referrer Blocker
RewriteEngine On
RewriteCond %{HTTP_REFERER} \.(af|al|at|au|be|bg|br|by|ch|cl|cn|cz|de|dk|ee|es|eu)(/|$) [NC]
RewriteEngine On
RewriteCond %{HTTP_REFERER} \.(fi|fr|gb|gr|hk|id|in|ir|it|jp|kr|kz|lt|lv)(/|$) [NC]
RewriteEngine On
RewriteCond %{HTTP_REFERER} \.(me|mu|my|mx|ng|pa|ph|pk|pl|ng|nl|no|ro|rs|ru)(/|$) [NC]
RewriteEngine On
RewriteCond %{HTTP_REFERER} \.(se|sg|si|sk|su|th|tn|tr|ua|ug|uk|uy|ve|vn|ye)(/|$) [NC]
RewriteRule .* - [F]
6:11 am on Mar 3, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


RewriteEngine On
is needed only once.

You can simplify the code...
RewriteCond %{HTTP_REFERER} \.(a[fltu]|b[egry]|c[hlnz]|d[ek]|e[esu])(/|$) [NC,OR]

The first three conditions need OR at the end.
8:14 am on Mar 3, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13256
votes: 361


Come to think of it, are there any countries you don't want to lock out?

RewriteCond %{HTTP_REFERER} \.[a-z][a-z](/|$)
RewriteCond %{HTTP_REFERER} !\.(c[ao]|nz)(/|$)
12:03 am on Mar 5, 2014 (gmt 0)

Junior Member

joined:Jan 12, 2014
posts:95
votes: 0


All but the USA ;)
12:23 am on Mar 5, 2014 (gmt 0)

Junior Member

joined:Jan 12, 2014
posts:95
votes: 0


I forgot to say thank to ALL for the advice 8)
2:29 am on Mar 5, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13256
votes: 361


All but the USA

If you mean it, then you really do want

RewriteCond %{HTTP_REFERER} \.[a-z][a-z](/|$)


to exclude all country codes everywhere. (What? Even .gl?!) I guess .us is theoretically possible, but they're surely up to no good.
3:22 am on Mar 5, 2014 (gmt 0)

Junior Member

joined:Jan 12, 2014
posts:95
votes: 0


OR

<Files>
deny from all
allow from .us
</Files>

All fun & games aside my main pests are .cn, .ru, .su, .de, .ua & .nl w/ webbost bots
7:20 am on Mar 5, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Are you sure it is REFERER you should be testing?