extending list of spiders to exlude

webtrends has a limit

[fom2001uk]

Anyone know how to amend webtrends' list of common soiders & robots. I've got an up-to-date list of spiders I want to exlude from a report, but I can't amend the list in webtrends. There's a limit on the list size. I need it to be a much bigger list.

Any way of doing this? (Anlayzer Series 7)

[cgrantski]

My spiders list is thousands long and goes into 14 jam-packed WebTrends custom hit filters. Just make new filters, call them "spiders 2" or whatever you wish.

[zgb999]

In an old WW post I found that the spider list in Webtrends is hardcoded so the only choice is to add a filter.

As more and more spiders are beeing built I would like to add the words bot and link to my spider filter.

Anybody knows of a legitimate browser (beeing not a spider) that I would exclude with those words?

[ogletree]

I just do an include with my stat program to show only Netscape and Internet Explorer. That helps stop most of them.

[fom2001uk]

I think you've cracked it there, ogletree. A great solution :-)

What about Opera though, market share still too small to worry about?

[ogletree]

Just look at your report before you do it see for yourself. Normaly it is way too small to worry about. It depends on your auidence. I bet Brett has a way above noraml percentage of non IE non Netscape users.

[fom2001uk]

Just checked the latest browser stats. Think I'll include Opera anyway (and Safari - now leader of the minority browsers). No big deal to include these also.

[fom2001uk]

Just tried that. Created an include only filter, including well known browsers (MSIE, Netscape, Safari, Opera).

But it hasn't knocked out all the spiders. Turns out a lot of spider visits are coming under Mozilla/5.0 (Yahoo Slurp is the culprite) and a few under Mozilla/4.0. So I can't separate the spider from the genuine browser using this method.

Ogletree, when you used this method, did you knock out Slurp visits, or did you have to add a separate exclude filter for Slurp?

[TheDoctor]

I'd include Konqueror as well, if you're going the "include" way. The less-used browsers may each provide a small proportion of the hits, but the figures can start to add up.

[ogletree]

I did a exclude * in my spider section but and did the include. It got rid of a lot of them but not all. I use web log expert.

[cgrantski]

I think you're circling back around to the brute force method: collect the IPs and/or User Agent strings of the zillions of known spiders and make more exclude filters.

[fom2001uk]

No chance!

I started doing that at first, and got as far as 20 filters (yes twenty) before I realised the size of the task. Even 20 filters only covered a small fraction of the list I've got (official robot exclusion list from a UK audit firm).

Unfortunately, Webtrends is what we're stuck with, and it doesn't let you just copy and paste a long list into one filter. I'm not doing 200 filters, so I'll stick with this include/exclude method.

Just done the Include (MSIE, Mozilla, Safar, Opera) and excluded Slurp, and it knocked out 98% of all spider visits. I can live with that. Hopefully my client can too :-)

[zgb999]

What about all those Java Browsers? Are they all spiders or are some legitimate ones around that should not be excluded?

[jaanis]

Where do you access the webtrends common spider list? I would like to add new ones.

[cgrantski]

It depends on the version of WebTrends that you use. Usually it's a Hit Filter, under the category "user agent" or "browser" or something like that. But if you tell me exactly which version you use I'll see if I can be more specific.

[jaanis]

I have WebTrends Analysis Suite 7

[fom2001uk]

It's under "Browsers" in the filter tab. Scroll right down the pull down menu till you get to common list of robots. You can edit it, but like I said earlier, there's very little room in the field box so you won't get many in there.

[jaanis]

I don't see the browser option anywhere

[jaanis]

OK I see the include option under filters, but isnt there a file I can edit and include the MSNBot?

[cgrantski]

There should be an "edit" button for every filter, but maybe they've locked this one. Just make another filter like it, put in your own strings, and make it global if you want it to apply to all your reports. Otherwise, edit each profile and activate the new filter. Would that work?

[jaanis]

how do i make it global?

[cgrantski]

If it's possible to make it global, there'll be a checkbox when you create the filter, or alternatively there'll be a checkbox next to the filter name on the master list of filters in the filter-creation section. I think with your version it's the latter.