Ignoramus Speaks

I have Webalizer and ...

[steverose]

I get a daily list of the top 15 user agents.

Yesterday in an affort to limit spidering to "good" spiders, I put up a robots.txt that conforms with the recommended one for WW.

When I vaidated it it had three errors which I cleaned up.

But when I looked at the Webalizer March 1 (day after)
list the results were depressingly familiar and I have no idea what they mean or who they are or how to proceed.

All help will be received with open arms. Best, S

Here's the first 15 user agents:

1 74 16.70% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1
2 65 14.67% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
3 28 6.32% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
4 20 4.51% Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
5 20 4.51% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProd
6 13 2.93% Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
7 13 2.93% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp 5.0.
8 12 2.71% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1
9 9 2.03% Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
10 9 2.03% Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FunWebPro
11 8 1.81% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4
12 7 1.58% Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)
13 7 1.58% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SEARCHALOT
14 6 1.35% Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)
15 6 1.35% Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)

[yowza]

Most of the bad bots ignore robots.txt. You have to block them with .htaccess.

[fiestagirl]

Most of those user agents look quite normal. No crawlers in there. It does look like the software is not showing you all of the characters in the user agent though.
What exactly worries you about this list?

[steverose]

Gosh, I thought I put in to get notifications of replies. Just returned to this.

Thanks for the responses.

The reason I was "disturbed" is because the Webalizer Report only shows 15 user agents -- the top -- and I would like to be aware of the "good" ones (Google, Yahoo, MSN, etc.).

Thanks for the heads up on the need for a htaccess -- something I knew about a few years back but guess I need a refresher course.

Cheers and thanks again,

S

[Damian]

Steve, do you have a file named "webalizer.conf" in your stats folder?
If you have you can edit it to say

AllAgents yes

Or add the line if it doesn't say anything about "AllAgents"
This option makes webalizer keep a list of all useragents, which you can access through a link under your top X overview. It is also possible for other items of the webalizer reports such as the searchterms and ip's.

If you do not have the file you can probably create it, depending on wheter your host allows it and is setup for it I guess. The webalizer site has examples of how to set up the webalizer.conf file.