Forum Moderators: phranque

Message Too Old, No Replies

Trying to figure out - need some help

         

macas

11:03 pm on May 20, 2011 (gmt 0)

10+ Year Member



Hello everyone on this part of forum.

During my problem with tons of web scrapers and hijackers I decide to make change in my .htaccess.
I have something like this and I want to add few more rules ...


<Files .htaccess>
order allow,deny
deny from all
</Files>
IndexIgnore */*

RewriteCond %{HTTP_HOST} ^www.website.com$
RewriteRule ^/?$ [website.comm...] [R=302,L]

ErrorDocument 404 [website.com...]

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !website.com [NC]
RewriteCond %{HTTP_REFERER} !google\. [NC]
RewriteCond %{HTTP_REFERER} !search\?q=cache [NC]
RewriteCond %{HTTP_REFERER} !yahoo\. [NC]
RewriteCond %{HTTP_REFERER} !bing\. [NC,OR]
RewriteRule \.(jpg|jpeg|png|gif|jpe|bmp|swf)$ [website.com...] [NC,R,L]
RewriteRule ^(.*)width=(.*)$ [website.com...] [R=301,nc,L]
RemoveHandler .html .htm
AddType application/x-httpd-php .php .htm .html


I just wanted to add few more rules like this :


RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^BadBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EvilScraper [OR]
RewriteCond %{HTTP_USER_AGENT} ^FakeUser
RewriteRule ^(.*)$ [go.away...]



RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteRule .* - [F]



order allow,deny
deny from 192.168.44.201
deny from 224.39.163.12
deny from 172.16.7.92
allow from all


I'm not too good with this and I don't know how add multiple "rewrite engine on " and other stuff, I really dont know how merge all this things together.

Can you give some thelp how to make this to work properly ?

Thanks,
Macas

wilderness

1:12 am on May 21, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



RewriteEngine on only takes place ONCE in your file.

There are many cautions expressed about the mixing of mod_alias and mod_rewrite, however I've done so for more than ten years.
Perhaps I've been quite lucky and utilized hosting companies which load the modules in the order function with my htaccess.

My order is the following:
1) mod_setenvif
2) mod_access
3) Rewrites

When using RewriteCond %{HTTP_USER_AGENT} it is necessary that you comprehend the use of anchors and the diversity of how they be applied.
1) begins with
2) ends with
3) contains
4) exactly as

Rather than listing each UA on a single line?
You may combine "like anchors" contained within parentheses and separated by the pipe character, which translates to "or"

Using your examples and utilizing the "begins with" anchor (^):
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC]

In the future, please use example.com in your coding and all the coding will remain visible.

macas

12:40 pm on May 23, 2011 (gmt 0)

10+ Year Member



Wilderness , thanks on good tips !

I'm now summarized all ideas and tips , can you tell me is this code correct :



IndexIgnore */*

Options +FollowSymLinks

RemoveHandler .html .htm
AddType application/x-httpd-php .php .htm .html

order allow,deny
deny from 224.39.163.12
deny from 172.16.7.92
# Block those annoying people, by denying a whole range of IP addresses.
deny from 192.168.
deny from 10.0.0.
# Block by country - COUNTRY_BLOCK_START
deny from 67.212.160.0/24
deny from 80.247.139.0/24
allow from all


<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /

#get rid of bad bots,Evil Scraper, hijackers
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC]
RewriteRule ^(.*)$ [go.away...]

#get rid of nasty websites
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteRule .* - [F]


RewriteCond %{HTTP_HOST} ^www.example.com$
RewriteRule ^/?$ http://example.com [R=302,L]

#Preventing hot linking of images and other file types
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !example.com [NC]
RewriteCond %{HTTP_REFERER} !google\. [NC]
RewriteCond %{HTTP_REFERER} !search\?q=cache [NC]
RewriteCond %{HTTP_REFERER} !yahoo\. [NC]
RewriteCond %{HTTP_REFERER} !bing\. [NC,OR]
RewriteRule \.(jpg|jpeg|png|gif|jpe|bmp|swf|css|js)$ http://example.com [NC,R,L]
RewriteRule ^(.)width=(.)$ http://example.com/ [R=301,nc,L]

</IfModule>


<IfModule !mod_rewrite.c>
# If we don't have mod_rewrite installed, all 404's
# can be sent to notfound.html, and everything works as normal.
ErrorDocument 404 http://example.com/notfound.html
</IfModule>

ErrorDocument 404 http://example.com/notfound.html



Thanks !

wilderness

1:29 pm on May 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Renove ALL open and closing <IfModule>

Once again, ONLY a single RewriteEngine on is used in the entire file.
Remove the other TWO.

order deny, allow
is the general consesus and more effective.

I've been using allow, deny for my decade, simply because "if it ain't broke", and it was I began using initially.


Eliminate these two redundant lines (unless you utilize different UA's):

RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC,OR]

Your also missing an escape line in the following:
RewriteCond %{HTTP_REFERER} !example.com [NC]

macas

2:10 pm on May 23, 2011 (gmt 0)

10+ Year Member



Ok , changed a bit.

Yes, I'll using for different UA's there , its just example.

BTW where is missing
' RewriteCond %{HTTP_REFERER} !example.com [NC] '
line exactly ?

Thanks a lot.



IndexIgnore */*

Options +FollowSymLinks

RemoveHandler .html .htm
AddType application/x-httpd-php .php .htm .html

order allow,deny
deny from 224.39.163.12
deny from 172.16.7.92
# Block those annoying people, by denying a whole range of IP addresses.
deny from 192.168.
deny from 10.0.0.
# Block by country - COUNTRY_BLOCK_START
deny from 67.212.160.0/24
deny from 80.247.139.0/24
allow from all

RewriteEngine on
RewriteBase /

#get rid of bad bots,Evil Scraper, hijackers
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(BadBot|EvilScraper|FakeUser) [NC]
RewriteRule ^(.*)$ [go.away...]

#get rid of nasty websites
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteRule .* - [F]

# Redirect from www
RewriteCond %{HTTP_HOST} ^www.example.com$
RewriteRule ^/?$ http://example.com [R=302,L]

#Preventing hot linking of images and other file types
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !example.com [NC]
RewriteCond %{HTTP_REFERER} !google\. [NC]
RewriteCond %{HTTP_REFERER} !search\?q=cache [NC]
RewriteCond %{HTTP_REFERER} !yahoo\. [NC]
RewriteCond %{HTTP_REFERER} !bing\. [NC,OR]
RewriteRule \.(jpg|jpeg|png|gif|jpe|bmp|swf|css|js)$ http://example.com [NC,R,L]
RewriteRule ^(.)width=(.)$ http://example.com/ [R=301,nc,L]

ErrorDocument 404 http://example.com/notfound.html

wilderness

6:20 pm on May 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Should read
!example\.com [NC]

macas

8:32 pm on May 24, 2011 (gmt 0)

10+ Year Member



Ok , got it !

Thanks a lot .