homepage Welcome to WebmasterWorld Guest from 54.227.12.219
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
CustomLog question: Apache or my own?
Use a PHP script, or let Apache do it?
craig1972




msg:3871880
 2:11 am on Mar 17, 2009 (gmt 0)

Hi.

I run several domains with WHM/Cpanel. I have currently disabled Apache customlog for my domains as I use google analytics JS includes.

But for just one domain I would like to enable CustomLog. From my reading of websites and apache's own mod_log_config ( [httpd.apache.org...] ) I think that this may be the best solution for me.

Question:

Is it worthwhile letting Apache do my logging and then have a PHP script that reads the logs periodically and cleans them up, or should I just do everything in PHP? My concern is that in the PHP scenario for each log I have to open and close a file handle. This is very quick these days, but is still an additional step. Or does Apache's own log also have to do this?

Thanks for any insight!

 

Caterham




msg:3872190
 12:14 pm on Mar 17, 2009 (gmt 0)

Or does Apache's own log also have to do this?

No, the logs are opened at startup/restart.

cleans them up

What do you need to clean-up? Use conditional logging if, e.g., images should not be logged.

craig1972




msg:3872496
 5:37 pm on Mar 17, 2009 (gmt 0)

I need to clean up because my boss wants my own report of access statistics, and all I can think of is to take some of this data into MySQL and then draw from there.

Once the logs are in there, I will empty the log file and have a clean new log file until I read them into MySQL again.

Is there a better way to do this? If I did not clean the log, wouldn't it become too big for accessing next time for the next report? The report needs to be live, so can be accessed any time.

Any thoughts welcome!

Caterham




msg:3872602
 7:19 pm on Mar 17, 2009 (gmt 0)

Is there a better way to do this? If I did not clean the log, wouldn't it become too big for accessing next time for the next report?

May be piped logging with the program rotatelogs [httpd.apache.org]. You can specify a file size as rotation time as well.

craig1972




msg:3872885
 2:41 am on Mar 18, 2009 (gmt 0)

Wow, thanks for this. It looks promising. I think Apache handling my log will surely be faster than opening/closing file handles in PHP.

But just to confirm this is feasible:

1. I only need to have this functionality for one domain in my Apache. That is, the default should be NO LOGS, but only inside one 'VirtualHost' definition for my target domain I need to have this customlog command. Is this possible?

2. Yes, I only need to log files coming from the main public_html folder. No images, and no other directories. How do I specify this in the command? The apache log page does not include examples of blocking .gif/.png/.jpg/.css/.js etc, or including only the root folder of the site.

3. Now if I understand the rotatelog thing, I can clean out the Apache log every day or so, and then read the "rotated" file using php into my database. Is this right?

Thanks!

Caterham




msg:3873183
 12:28 pm on Mar 18, 2009 (gmt 0)

but only inside one 'VirtualHost' definition [...] Is this possible?

Yes.

I only need to log files coming from the main public_html folder. No images, and no other directories.

Do you have images within the root folder as well?

# match the root folder (all requests, subfolders/files in subfolders won't be matched)
SetEnvIf Request_URI ^/[^/]*$ log
#
# *alternatives* with possessive quantifiers and look-around assertions
# match the root folder, but no images within that folder (subfolders/files in subfolders won't be matched)
# requires apache2.0+
# SetEnvIf Request_URI ^/[^/]*+(?<!\.jpg¦\.gif¦\.css¦\.js)$ log
# *or*
# match the root folder, if it's easier to specify allowed extensions
# instead of excluding some (subfolders/files in subfolders won't be matched)
# requires apache2.0+
# SetEnvIf Request_URI ^/[^/]++((?=\.html$¦\.jsp$)¦$) log
#
CustomLog "¦bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" combined env=log

-or- the opposite way: Excluding everything which shouldn't be logged

SetEnvIf Request_URI ^/[^/]+/ dontlog
SetEnvIf Request_URI ^/.++(?<=\.jpg¦\.gif¦\.css¦\.js)$ dontlog
CustomLog "¦bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" combined env="dontlog

3. Now if I understand the rotatelog thing, I can clean out the Apache log every day or so

The log rotates automatically.

You'll have to replace all ¦ pipes with solid ones since the latin1 character 7C is replaced with A6 when posting to this forum.

jeffatrackaid




msg:3873234
 2:12 pm on Mar 18, 2009 (gmt 0)

craig1972
You said you were using cPanel. It should handle log rotation automatically. If it is not, then something has gone haywire, which is not uncommon with cPanels log rotation.

You may want to check the cpanel forums for assistance. I don't recall off the top of my head how you reset log rotation on cpanel for the virtual sites.

Logs Are Important

Logs are important for security and diagnostic reasons. Recently, I had a new client who had disabled their access logs because they were bogging down the disk. They were hacked. I had no way of determining the source of the hack since there were no logs. We were forced to put the system back in place and monitor it. Within 24 hours they were attacked again, but this time, we had the logs and identified the insecure scripts.

craig1972




msg:3873557
 6:32 pm on Mar 18, 2009 (gmt 0)

Thanks Caterham. That is really kind.

I have Apache 2 installed. Does rotatelog come preinstalled with it, or will I have to install it? There's no "bin/rotatelog" yet.

Also, in this code:


SetEnvIf Request_URI ^/[^/]+/ dontlog
SetEnvIf Request_URI ^/.++(?<=\.jpg¦\.gif¦\.css¦\.js)$ dontlog
CustomLog "¦bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" combined env="dontlog"

Note the last doublequote I have added. Is this right?

And where in this code have I specified what I need to store...there is no "LogFormat" command. Do I need to add that too in my VirtualHost definition? I would like to store it in this format:


<Time>¦¦<FileName(before Redirect, if redirect)>¦¦<Referer>¦¦<IP Address>

craig1972




msg:3873591
 7:04 pm on Mar 18, 2009 (gmt 0)

Does this make sense:

[quote]
SetEnvIf Request_URI ^/[^/]+/ dontlog
SetEnvIf Request_URI ^/.++(?<=\.jpg¦\.gif¦\.css¦\.js)$ dontlog

//---------------FORMAT-----------------
//<Time>¦¦<FileName(before Redirect, if redirect)>¦¦<Referer>¦¦<IP Address>¦¦MyDomain
//--------------------------------------

LogFormat "%!404,500%t¦¦%U¦¦\"%{Referer}i\"¦¦%a¦¦%v"
CustomLog "¦bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" combined env="dontlog"
[/quote]

Caterham




msg:3873676
 8:29 pm on Mar 18, 2009 (gmt 0)

There's a typo in my posting, I have hit the wrong key, env="dontlog should read env=!dontlog

You should specify a name for your LogFormat which could be referenced in your special log file (instead of combined).

LogFormat "%!404,500%t¦¦%U¦¦\"%{Referer}i\"¦¦%a¦¦%v" speciallog
CustomLog "¦bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" speciallog env=!dontlog

The support programs should be compiled as well. Do you have ab, apxs, htpasswd etc?. You can link the program statically with --enable-static-rotatelogs

craig1972




msg:3873909
 1:51 am on Mar 19, 2009 (gmt 0)

Thanks. Linking statically means I need to recompile Apache with that option? Is there a way to enable rotatelogs without recompiling? Thx

craig1972




msg:3878604
 5:18 pm on Mar 25, 2009 (gmt 0)

Ok, I have this working finally. Thanks so much for the help!

My one last question: is it conditionally possible to store only those URLs which satisfy a condition which I can only check by making a trip to the MySQL database? Currently my conditions of skipping GIF or JPG files for example are working great (thanks!) but I also need to check another thing about the URL from the database. Any way to do that?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved