homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: DixonJones & mademetop

Website Analytics - Tracking and Logging Forum

Tracking spiders w/javascript based tracking tool
on a very large site

10+ Year Member

Msg#: 3499449 posted 9:18 am on Nov 8, 2007 (gmt 0)

We have a large web site with over 500,000 UV a day and the raw log file is over 1GB after zipping.

We are considering using a javascript based web analytics but want to detect spiders/bots to determine whether to block them or not.

It would be painful to run a separate log analyzer manually with that size of log file.

Currently we are using a log file based analysis solution, so we know there are too many malicious bots to block.

How do we know after installing this javascript based analytics if spiders with or without USER_AGENT visit our web site? What should we do to track some of the spiders and block them?


Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved