What WC3 properties to track in my logs if space is an issue

[jacqueschoquette]

Hi I just started working for a company that is growing like crazy. So fast that they are quicly outgrowing there server hardware.

The log files on the site were getting out of hand and talking up all the much needed space on the server so prior to my arrival they were deactivated.

I know that some of this information is much needed and am trying to find a happy medium on what information to track. I am expecially interested in seach engine spider activity and trying to find out what pages on the site are getting index or not.

Can somebody tell me what are the key wc3 extended logging properties to collect in the log files for the purpose of studying spider activity.

Also are there any tricks that can be used to minimize the size of the log files generated. Like possibly avoiding logging of downloaded images and things like that. What log properties are just plan useless and could be unchecked to save space.

Thank you for you assistance

[daveVk]

A better aproach may be to process log files more frequently, and archive log files after processing. Also look at getting more storage its cheap compared to value of log files.

[nmattheij]

Set up an automatic system that:
- Zips log files
- Copies them to another server with sufficient disk space
- Deletes original log-files after x days or even hours

[cgrantski]

Not logging images, css, and js will reduce the size of your logs by 90% or more, in most cases.

Zipping will reduce them by 90%.

90% x 90% means a 99% reduction in size if you do both.

There are very few log fields you can live without, and the ones you can live without are really short fields.

[jacqueschoquette]

how can i prevent css and images and js from not being logged i have read this before but i do not know where to set this can sombody please advise?

[gregbo]

Logging images, css, and js may help you to determine whether you have fraudulent traffic (if the fraudsters aren't smart enough to make those types of requests). IMO, what you should try to do is determine what your business requirements are, reduce the logs to those requirements before putting them in your data warehouse or whatever you're using for analysis, and then use the cheapest mode of long-term archival available to you for offline storage. Your business requirements will also determine what specfically you need to log in the first place, e.g. cookies, referrers, requested pages, etc.

[cgrantski]

Yes and logging images also is a cheap way to find missing images.

However if you do want to turn off logging for anything, in IIS it has to be done for whole directories, and won't work if your images etc are not in their own folder. Just go to the folder and edit the Properties; there will be a checkbox for logging.