Web Trends - Excluding bots

[jaime1985]

Hi everyone,

I'm fairly new to all this webstats reporting, but I've come to realise just how important it is :-)
So I've started using an old version of WebTrends Log Analyzer 6.5C, and I need to know how to exclude bots, robots, spiders (whatever you wanna call them) from being listed as hits pageviews on my website.
WebTrends seems to have an inbuilt Exclude filter for this kind of thing, but it doesnt seem to exclude many of the popular bots etc. This may be due to my version of WebTrends being out-of-date (I would use WebTrends but we dont have a machine powerful enough:-(!)

Is there a way to guarantee I can omit these bots and spiders from my report? If there is I'd sure like to know how! Is it just a matter of updating a list from somewhere?

Thanks for anything

James

[nmattheij]

In the new version there's a file called 'browser.ini' in which all browser and spider definitions and client strings are defined.

If I'm not mistaken, this file is also available in older versions?

You can add spiders to the file and the [spiders and bots] exclusion filter will exclude them.

Obviously this will take some work, but maybe there are standards lists somewhere out there.

[jaime1985]

Unfortunatly this file doesn't exist! :-(...Thanks anyway!

Any other ideas? anyone?
lol

James

[cgrantski]

I think it's browsers.ini not browser.ini and I'm pretty sure it has existed in all versions of WT since the dark ages, so check again (installation root directory is where it usually is).

But your best bet by far is to leave browsers.ini alone and make your own filters. Base them either on address (IP) or browser (strings found in the User-Agent field such as "googlebot" or "bot" etc) and turn them on for all your profiles. You can put lots of IP addresses etc into one of these filters; I think the limit is 256 characters, with IP addresses separated by spaces when you enter them in the WT interface. Bear in mind that for IP ranges WT is most reliable if you give it CIDR notation for ranges - instead of "0.0.0-25.0-255" use 0.0.0.0/19 or whatever the CIDR notation would be. You can find CIDR submask converters on the net.

With just a little research you can exclude the vast majority of all bots, spiders etc with just a few filters that focus on the UA field. With a lot of effort you can get rid of nearly all of them, but it can be a lot of work. Our spider/bot filters fill up about 60 WebTrends filters, and we find new ones almost every time we examine the visitor tables.

Finally, I think WT issues updates of browsers.ini somewhere on their web site support area.3