How do you determine your daily hits?

Do you forget the spiders?

[Googly]

I'm using a log analysis tool at the moment 'WebLog Expert Lite'. I'm trying to gather a general idea of how many different users enter my site per day. I kinda assume these are unique IPs. The thing is it seems obvious that a certain amount are search engine spiders.

How do you guys seperate the spiders from the human users?

I really haven't got time to do a WHOIS on each IP to see where it originates. I assume the only way is to take an 'average amount of spider entries per day' figure and take it away from the overall visits. But is there really an average amount? I don't think so, it fluctuates so much.

How do you do it?

Cheers
Googly

[Travoli]

Honestly, I just count spiders into the unique visitors #. I have software that picks SE spider IP's out and lists them seperately. Looking right now, only, there were only 100 unique SE spider IP's over a 3 month period, and with thousands of visitors, it doesn't skew stats enough for me to worry about.

[chiyo]

I would tend to agree with Travoli. If you are getting more than a few thousand uniques a day, its not going to make much difference to the already very fuzzy data. If you are looking at a whole month its insignificant. As long as you are looking at unqiues only, cause googlebot, as long as you have cnfigures all the diff types, are counted as 1 visitor.

However if you are counting daily uniques and have less than 1,000 a day you may find that even 50 of these may be spiders that dont really read the page. Not only SE bots, but email address scrapers, and heaps of other automated bots, increasing all the time. Certainly you should count only those pages which read the whole page, and not just the headers as well.

This is one reason why Page views can be such a rubbish statistic - our daily crawls from googlebot, news services accessing our rss files and scraping content, and the like account for sometimes 10% or more of our daily "page views" so we do filter them out.

[turk182]

I agree with Travoli and Chiyo. There are not so many spiders that they may significally affect you statistic of different users. At worst, they may affect the number of viewed pages, but this is also not a big problem. I've found that in the most spidered days the page views of that spiders were about a 4% of our daily transit.