homepage Welcome to WebmasterWorld Guest from 54.83.133.189
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
.htaccess ban bad bots ...
.htaccess ban bad bots ...
StopSpam




msg:1509394
 2:50 pm on Aug 17, 2003 (gmt 0)

.htaccess ban bad bots ...

if i want to ban bad bots with my htaccess file should i use this or?

i want them to banned in al sub folders as well ...

Order Allow,Deny
Allow from all
Deny from env=bad_bot

or

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

or

<Files ~ "^.*$">
order allow,deny
allow from all
deny from env=bad_bot
</Files>

or

<Files ~ "\.(shtml¦SHTML¦htm¦HTM¦html¦HTML)$">
order allow,deny
allow from all
deny from env=bad_bot
</Files>

i dont know wish one to use for auto ban in all sub folders on my server, imgetting cufused here hehehe

they all look very simular but i am sure they all do something different

 

claus




msg:1509395
 2:59 pm on Aug 17, 2003 (gmt 0)

1) <Limit GET POST>

Bans everything that uses one of these two request methods (GET or POST) no matter which file. Does NOT ban HEAD, PING, and other methods.

2) <Files ~ "\.(shtml¦SHTML¦htm¦HTM¦html¦HTML)$">

Bans all requests that have one of these file endings. Does NOT ban ".gif" ".jpg" ".asp" ".php" or others.

3) <Files ~ "^.*$">

Bans any kind of request for any file, no matter what the name is.

Number three will ban everything, including "robots.txt" and all pictures and scripts.

4) Order Allow,Deny

The first one you mentioned works exactly like number 3, it bans all, it's just another way of doing it.

/claus

StopSpam




msg:1509396
 4:02 pm on Aug 17, 2003 (gmt 0)

Hi Claus thx

i think i go for the :

Order Allow,Deny
Allow from all
Deny from env=bad_bot

its most understandable code
so your sure this bans also automaticly in all sub directories if i place the code in base or root or ww dir?

claus




msg:1509397
 4:14 pm on Aug 17, 2003 (gmt 0)

>> this bans also automaticly in all sub directories

Yep. Unless you put in another .htaccess in one of the subdirs that allows something this one bans. If you just have one .htaccess-file in root the things you write in this one will also work on subdirs.

/claus

StopSpam




msg:1509398
 4:23 pm on Aug 17, 2003 (gmt 0)

i have a second folder that use a .htaccess file its a password protected folder using this code;

and i want to be sure bad bots or users are banend here as well even if they have correct login/pass

AuthUserFile /usr/www/public/html//cgi-bin/.htpasswd
AuthGroupFile /dev/null
AuthName "Private pdf files"
AuthType Basic
Satisfy Any

<Limit GET POST>
order deny,allow
deny from all
require valid-user
</Limit>

wish code is compatable with above
so if one gets banned he is banned inthe password folder to? is that posible? i do not wish towrite the ban to password htaccess

claus




msg:1509399
 3:01 pm on Aug 18, 2003 (gmt 0)

StopSpam, this will quickly become complicated, i'll try to take it step-by-step. If i do make errors and it does not work.. well, we'll just have to sort it out somehow.

This is your root .htaccess:

Order Allow,Deny
Allow from all
Deny from env=bad_bot

Actually you only need the line "Deny from env=bad_bot", but i suggest you try this in stead, it's a very simple one

<Files *>
Deny from env=bad_bot
</Files>

The asterisk (*) is a wildcard for "any sequences of characters", it's the same as writing this:

<Files ~ "^.*$">

Note that you don't really need the "Order" or the "Allow" if all you want to do is to "Deny" a specific User-Agent. But putting it inside the <Files> will give you a hint what it's there for, although these can also be omitted. You'll see below why i recommend them.

Now for the passwords:



Access control using two .htaccess files:

What you are trying to do can be done using just the one root .htaccess file, but you asked for two, so i'll explain it using two first, and then using one after. There's a couple of things to remember when working with two or more .htaccess files. The first is:

If you can even make an .htaccess and get it to work, then your server has the "AllowOverride" directive set. This means that whatever you do with an .htaccess file in a subdirectory will override what has been written in higher directories.

so, you have to include all the commands from your /root/ .htaccess in your /subdir/ .htaccess if you want them to work for the /subdir/ as well. Otherwise they will be overrided.

This means that you can have this:

[x]
¦
¦--[y]
¦ ¦
¦ ¦--[z]

Where directory [x] is open for some ("not bad-bot"), directory [y] is access restricted (username:password), and directory [z] is open for all (all bots and no passwords) - all this takes is three .htaccess files, one for each directory.


The password restriction:

This is the easy part. In the "protected-directory" .htaccess file, write this:

--------------------
AuthType Basic
AuthUserFile /usr/www/public/html/cgi-bin/.htpasswd
AuthName "Private pdf files"
Require valid-user
--------------------

These four lines are the AuthConfig commands, less the "group" thing which you don't need (it was set to /dev/null/ which is the trashcan or nothing).

I've also deleted "Satisfy Any" as the default ("Satisfy All") is what you need. Satisfy is used when you both have "Allow" and "Require" and it decides if one or both should pass the check. I know that i deleted the "Allow" and now there's only a "Deny" but in this respect the two should be equal.

You want both checks to pass ("not bad-bot" AND "valid user") so it must be All, which is the default.

Add to this /subdir/ .htaccess file all commands from your /root/ .htaccess that you want to use for the /subdir/ also. Otherwise they will not work here.


Alternative method, using one .htaccess file:

Here you only need your /root/ .htaccess. You still need the .htpasswd file but you don't need the /subdir/ .htaccess file.

You need to include the AuthConfig statements from above, but inside a set of <Files></Files>, like this:

-----------------
<Files /usr/www/public/html/password-protected-dir>
AuthType Basic
AuthUserFile /usr/www/public/html/cgi-bin/.htpasswd
AuthName "Private pdf files"
Require valid-user
</Files>

-----------------

I think this should work for you. It's also easier, as you only need one .htaccess file.

As this is enclosed in <Files> and you also have the other <Files> restriction (above, top of post) you can easily see which settings is for which files on your server using only one .htaccess file.

The "Satisfy All" is still there by default, so the server will know thet the AuthConfig rules are not the only ones that must pass, the "bad-bot" rules must also pass.

I hope you can get this to work - please post if it works and of course also if it gives any problms.

/claus


added:

<Limit> or <Files>?

I recommend always using <Files> in stead of <Limit>. The <Limit> command does not do what one thinks it does. As an example, see this:

<Limit GET>
Deny from 127.0.0.0
</Limit>

This command will deny the IP 127.0.0.0 from doing a GET request. It will not deny the IP from doing a POST request, not a DELETE either.

Here's a list of methods for HTTP 1.1:

GET, HEAD, POST, PUT, DELETE, OPTIONS, TRACE, CONNECT (reserved)

HTTP 1.0 may have other methods, and future versions will probably have theirs as well. Clearly, if you Deny with <Limit GET>, then you also Allow the others.

If, you use <Files> the method does not matter. Example:

<Files *>
Deny from 127.0.0.0
</Files>

Will deny any attempts by 127.0.0.0 to do anything with any file, no matter if it is GET, POST, HEAD, PUT, ...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved