PeterD - 5:51 pm on Aug 3, 2003 (gmt 0)

Yeah, I use both BROWEXCLUDE and HOSTEXCLUDE, but I rely more on BROWEXCLUDE. (It looks to me like spoofed UAs make up only a tiny fraction of total traffic, although there are occasionally obvious spiders with faked or empty UAs that you have to filter out by IP address or domain name).

I don't know where you can download a complete list--I have just built up my own over time. Usually I run Analog, look through the Browser Summary for UAs I don't like, copy them from the report and paste them right into the exclude list, then re-run Analog.

Over the last year, I've added about 200 entries to my exclude list (which, for convenience, is in a separate config file that I call from the main config file).

I'm sure I'm missing some low volume bots that grab one or two pages here and there, but the big ones make up most of the spider traffic anyway. The little ones are just statistical noise. Server log file analysis is an inherently imperfect exercise anyway!

Hope this helps,

Pete