Identifying spiders using WebTrends Live

How does one do it?

[GraemeV]

I was wondering if anyone knew how to use WebTrends Live to see which spiders have been visiting a site?

Webalizer has the "Agents" category, where you can see which spiders have visited recently.

Does WebTrends Live have something similar?

[biggles]

WebTrends Live is a remotely hosted traffic tracking solution that uses a snippet of JavaScript embedded in web pages to track visitors. Trouble is JavaScript is ignored by search engine spiders so they don't get tracked. You need to examine your web server log files to see spider visits (Webalizer is a webserver log analysis package which is why it can show you spider visit details).

An alternative approach you might consider is installing GoogleStats, a free Open Source PHP application for analyzing spider visits to a site. Details here [googlestats.com ]. Haven't tried it myself yet but it looks good. Maybe others have tried it & can comment.

Hope that helps.

[Alternative Future]

Hi GraemeV and welcome to WebMasterWorld,

Also another good utility is available @ google-dance dot com this one uses perl.

HTH,

-gs

[sharpie]

I am going to use webtrends log analyser to track spider activity on the sites that i manage. Has anyone got any tips about setting up spider tracking.

I have seen lists of spider ip addresses listed elsewhere and intend to use those.

I suppose what i'm saying is what should i be looking for and can anyone save me wasting loads of time looking for the wrong stuff? What does everyone else do?