Search Engine Bots

Do you include in the statistics?

[Maynard]

MSN, Google and Inktomi robots spider our site every single day - surely their presence on our site artificially inflates the web statistics. Do you subtract them from your figures when you collate unique visitor numbers, session times, geographic locations etc?

Thanks,
Maynard.

[bufferzone]

I don’t remove the hits from the statistics, but in my rapport where I break down the statistics, I take them into account. When dealing with sales and customer prognoses the bots are removed, but in my estimates on bandwidth usage they are a part of the numbers of course.
I think the point her is, that statistics are just statistics and they need to be digested and fermented by you. Apply your intelligence and present the data in a meaningful way. Sometimes the bots need to be a part of the equation and sometimes not. It depends on what your are looking at.

[Maynard]

Thanks for your comments. We give preference to counting total visitors based on individual IP addresses. I take it that Googlebot has its own IP address (or a range of 1-18) meaning that even though it visits us everyday (many times of the day) our unique visitors total will only count 1-18 (IP address)?

[larryn]

In our stats, we segregate the bots and the hit & run visitors, that way we can concentrate on the 'real' visitors seperately from the bots (who get their own review) and hit & runs (where we try to figure out why the clicked on our site at the search portal, and if we care).

It all depends upon your purpose - if you are looking for bandwidth (for example), every byte counts; whereas if you are looking at usage, you need to determine who is your target.

[pageoneresults]

We're using two forms of statistics. One is web based and the other is server based. The client only reviews the web based statistics. We have filters in place for the web based statistics to exclude all development personnel along with employees within the client organization. This provides by far the most accurate statistics based on the tools available to us.