Welcome to WebmasterWorld Guest from 34.229.24.100

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Ignoramus Speaks

I have Webalizer and ...

     
8:43 pm on Mar 1, 2004 (gmt 0)

Full Member

10+ Year Member

joined:May 7, 2003
posts:260
votes: 0


I get a daily list of the top 15 user agents.

Yesterday in an affort to limit spidering to "good" spiders, I put up a robots.txt that conforms with the recommended one for WW.

When I vaidated it it had three errors which I cleaned up.

But when I looked at the Webalizer March 1 (day after)
list the results were depressingly familiar and I have no idea what they mean or who they are or how to proceed.

All help will be received with open arms. Best, S

Here's the first 15 user agents:

1 74 16.70% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1
2 65 14.67% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
3 28 6.32% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
4 20 4.51% Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
5 20 4.51% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProd
6 13 2.93% Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
7 13 2.93% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp 5.0.
8 12 2.71% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1
9 9 2.03% Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
10 9 2.03% Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FunWebPro
11 8 1.81% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4
12 7 1.58% Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)
13 7 1.58% Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SEARCHALOT
14 6 1.35% Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)
15 6 1.35% Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)

11:44 pm on Mar 2, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 30, 2003
posts:430
votes: 0


Most of the bad bots ignore robots.txt. You have to block them with .htaccess.
10:24 pm on Mar 4, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


Most of those user agents look quite normal. No crawlers in there. It does look like the software is not showing you all of the characters in the user agent though.
What exactly worries you about this list?
6:25 pm on Mar 5, 2004 (gmt 0)

Full Member

10+ Year Member

joined:May 7, 2003
posts:260
votes: 0


Gosh, I thought I put in to get notifications of replies. Just returned to this.

Thanks for the responses.

The reason I was "disturbed" is because the Webalizer Report only shows 15 user agents -- the top -- and I would like to be aware of the "good" ones (Google, Yahoo, MSN, etc.).

Thanks for the heads up on the need for a htaccess -- something I knew about a few years back but guess I need a refresher course.

Cheers and thanks again,

S

1:51 pm on Mar 6, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 19, 2000
posts:366
votes: 0


Steve, do you have a file named "webalizer.conf" in your stats folder?
If you have you can edit it to say

AllAgents yes

Or add the line if it doesn't say anything about "AllAgents"
This option makes webalizer keep a list of all useragents, which you can access through a link under your top X overview. It is also possible for other items of the webalizer reports such as the searchterms and ip's.

If you do not have the file you can probably create it, depending on wheter your host allows it and is setup for it I guess. The webalizer site has examples of how to set up the webalizer.conf file.