Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Tracking spiders w/javascript based tracking tool

on a very large site

9:18 am on Nov 8, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:June 13, 2003
posts: 379
votes: 0

We have a large web site with over 500,000 UV a day and the raw log file is over 1GB after zipping.

We are considering using a javascript based web analytics but want to detect spiders/bots to determine whether to block them or not.

It would be painful to run a separate log analyzer manually with that size of log file.

Currently we are using a log file based analysis solution, so we know there are too many malicious bots to block.

How do we know after installing this javascript based analytics if spiders with or without USER_AGENT visit our web site? What should we do to track some of the spiders and block them?