search engine spiders and log files

[valeria_vi]

i am reading thsi stuff about disadvantages of log analyzing and here's one of teh things written:
"[se spiders/robots] skew the statistics gathered by log-based systems because they are counted right along with real browser hits."
is it true? and if so, how do you deal with it?

[incywincy]

usually logfile analyzers include some form of configuration whereby you can exclude user agents or ip addresses. at first this involves a fair amount of time to set up but once it is up and running it is then an iterative process of viewing the reports and then adding unwanted user agents and/or ip addresses to the config file to be ignored on future reports.

personally i generate 2 sets of logfile analysis reports, one for human visitors (spiders and other bots excluded) and one purely for googlebot.

[lazerzubb]

Good logtools, filter this out automatically, or they present it in a seperate file, and exclude it from the main one.

A really good thread about what to use in your logfiles and what to exclude [webmasterworld.com]

[valeria_vi]

"usually logfile analyzers include some form of configuration whereby you can exclude user agents or ip addresses." - this is what i thought to myself as well.
lazerzubb, - great thread link.
thanks a lot to both of you!