Welcome to WebmasterWorld Guest from 107.20.5.156

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Tracking spiders w/javascript based tracking tool

on a very large site

     

itisgene

9:18 am on Nov 8, 2007 (gmt 0)

10+ Year Member



We have a large web site with over 500,000 UV a day and the raw log file is over 1GB after zipping.

We are considering using a javascript based web analytics but want to detect spiders/bots to determine whether to block them or not.

It would be painful to run a separate log analyzer manually with that size of log file.

Currently we are using a log file based analysis solution, so we know there are too many malicious bots to block.

How do we know after installing this javascript based analytics if spiders with or without USER_AGENT visit our web site? What should we do to track some of the spiders and block them?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month