itisgene - 9:18 am on Nov 8, 2007 (gmt 0)

We have a large web site with over 500,000 UV a day and the raw log file is over 1GB after zipping.

We are considering using a javascript based web analytics but want to detect spiders/bots to determine whether to block them or not.

It would be painful to run a separate log analyzer manually with that size of log file.

Currently we are using a log file based analysis solution, so we know there are too many malicious bots to block.

How do we know after installing this javascript based analytics if spiders with or without USER_AGENT visit our web site? What should we do to track some of the spiders and block them?