Is there a way to unclutter my logs?

Forum Moderators: phranque

Message Too Old, No Replies

Is there a way to unclutter my logs?

littleman

2:56 am on Nov 2, 2001 (gmt 0)

Like the rest of you, I am getting tens of thousands of worm requests cluttering my logs. Anyone know how to get apache to not log errors?

littleman

3:06 am on Nov 2, 2001 (gmt 0)

Basically what I'm trying to do is keep all the info in a 'combined' log but exclude all the 404's. I know I could inverse grep the logs and remove them that way, but I was hoping for something more elegant.

Gorufu

9:43 am on Nov 2, 2001 (gmt 0)

Logging of trash or virus related requests can be prevented in the latest versions of Apache using the mod_setenvif module if it is enabled.

You will need to have access to httpd.conf and it will work with Apache versions 1.3.13 and later. It should with Apache 1.3.7 and later.

Basically you prevent logging of specific directories using SetEnvIf and/or SetEnvIfNoCase

Enter the following in httpd.conf or srm.conf
I use srm.conf because it is less cluttered and easier to edit.

SetEnvIf Request_URI \.ida trash
SetEnvIf Request_URI /scripts trash
SetEnvIf Request_URI /c/winnt trash
SetEnvIf Request_URI /d/winnt trash
SetEnvIf Request_URI /_mem_bin trash
SetEnvIf Request_URI /_vti_bin trash
SetEnvIfNoCase Request_URI /msadc trash

It will also work with User-agents
SetEnvIf User-Agent NetMechanic trash

An environment variable trash is assigned to all requests for the above directories and files. To prevent logging you add env=!trash to your CustomLog path for each virtual host.

CustomLog /home/vhost/logs/access.log combined env=!trash

Apache will need to be restarted to effect the changes.

Also [url=www.webmasterworld.com/forum13/682.htm]BrowserMatch[/url] is an effective way of blocking bandwith sucking and pesty bots on a server wide basis.

littleman

12:18 am on Nov 3, 2001 (gmt 0)

Hey, that will work. Thanks!

Gorufu

12:03 am on Apr 4, 2002 (gmt 0)

Recently I noticed that all the Nimda requests end with either
root.exe or cmd.exe


SetEnvIf Request_URI \.ida  trash 
SetEnvIf Request_URI /scripts trash  
SetEnvIf Request_URI /c/winnt trash  
SetEnvIf Request_URI /d/winnt trash  
SetEnvIf Request_URI /_mem_bin trash  
SetEnvIf Request_URI /_vti_bin trash  
SetEnvIfNoCase Request_URI /msadc trash

can be replaced with


SetEnvIf Request_URI (.*)cmd\.exe trash 
SetEnvIf Request_URI (.*)root\.exe trash 
SetEnvIf Request_URI \.ida(.*)$  trash

mivox

12:22 am on Apr 4, 2002 (gmt 0)

For the less technically inclined, some statistics/log analysis scripts include a log cleaning function. Accessprobe is one I've used... I set it to delete every request containing ".exe" right now, and my logs stay quite small, without losing any "real" 404 data.

Searching for "access log cleaning script" or something along those lines might turn up something else useful. :)

ineedhelp

4:19 pm on May 5, 2002 (gmt 0)

[root@host conf]# service httpd restart
Shutting down httpd: [ OK ]
Starting httpd: Syntax error on line 4 of /etc/httpd/conf/httpd.conf:
Illegal option SetEnvIfNoCase
[FAILED]
Syntax error on line 4 of /etc/httpd/conf/httpd.conf:
Illegal option SetEnvIfNoCase
[FAILED]
Syntax error on line 4 of /etc/httpd/conf/httpd.conf:
Illegal option SetEnvIfNoCase
[FAILED]
[root@host conf]#

ineedhelp

4:58 pm on May 5, 2002 (gmt 0)

Well thats accessprobe out,
its totally useless.

Geesh I have never seen
such a piece of trash before.

brotherhood of LAN

8:25 pm on May 5, 2002 (gmt 0)

Thanks mivox, that sounds more on my level ! :)

I'll be stickying littleman in a few months when im there ;)

littleman

8:34 pm on May 5, 2002 (gmt 0)

ineedhelp, put the routine below the section where apache loads the modules. A good place to put it is above the section where you list your virtual hosts. I suspect that apache is erroring out because you placed that routine before the modules are loaded.

ineedhelp

11:45 pm on May 5, 2002 (gmt 0)

Ok they were put after the
modules and before the virtual
servers.

[root@host conf]# service httpd restart
Shutting down httpd: [ OK ]
Starting httpd: [FAILED]
[FAILED]
[FAILED]
[root@host conf]#

jdMorgan

4:50 am on May 7, 2002 (gmt 0)

Apache version problem?

From Apache documentation:
----
SetEnvIf directive

Syntax: SetEnvIf attribute regex env-variable[=value] [env-variable[=value]] ...
Default: none
Context: server config, virtual host, directory, .htaccess
Override: FileInfo
Status: Base
Module: mod_setenvif
Compatibility: Apache 1.3 and above; the Request_Protocol keyword and environment-variable matching are only available with 1.3.7 and later; use in
.htaccess files only supported with 1.3.13 and later

Gorufu

6:28 am on May 7, 2002 (gmt 0)

Hi ineedhelp

> Ok they were put after the modules and before the virtual servers.

> Shutting down httpd: [ OK ]
> Starting httpd: [FAILED]
> [FAILED]
> [FAILED]

The above error is strange. No reason was given as to why Apache failed. There should be an entry in your access_log and/or /var/log/messages file giving a reason why Apache failed. It could be a CustomLog file related error.

> Apache version problem?

You probably have Apache 1.3.7 or later because you did not get the Illegal option error again.

> Override: FileInfo

Check httpd.conf to see if AllowOverride is set to All or at least contains FileInfo

<Directory />
AllowOverride AuthConfig Limit FileInfo Options
Options All
</Directory>

Check you sticky mail

ineedhelp

3:32 pm on May 7, 2002 (gmt 0)

Sorry!

Forgot to list my apache version:

Server version: Apache-AdvancedExtranetServer/1.3.23 (Mandrake Linux/4mdk)
Server built: Mar 4 2002 21:08:36