Raw logs vs. customized logs (via CGI)

advantages and disadvantages

[alexjc]

I've been hesitating about this for a while. It's fairly difficult to extract exact information from raw logs (like unique visitors per day, see other threads), but as a webmaster you need to setup absolutely nothing to get this working.

For customized logs, you obviously need to run CGI for each page, and write to your own log file. This way you can easily track independant users with cookies, and all their session information: pages visited, links followed...

What do the rest of you use? Are there any alternatives i haven't looked into??

Alex

[oilman]

hi alex - welcome to WebmasterWorld.

raw logs are definitly a chore to wade through but you don't necessarily need a custom cgi script to handle logging for you. There are many good programs that you can buy (some are free) that will filter and analyze the raw logs for you.

Check out this thread for some suggestions:
[webmasterworld.com...]

[spiky]

Could some one cover some basics for me. what is a unique visitor ( and may be a few other definitions of the data these logger tools produce. )

Is there a basic tutorial out there ??

Thanks

[oilman]

here's some basics. These are the two most confused for each other.

unique visitor: can also be looked at as 'unique ip' and pretty much means individual visitors. This number strips out multiple visits from the same IP over the given reporting period. For example: even if some one visits that site 50 times in one day they will only be counted as 1 unique visitor for that day.

hits: this is the most commonly misrepresented term online. The true definition is that hits refers to the number of times the server is 'hit' for a file call. If you have a 6 images on a page and a surfer goes to that page the server will record 7 hits - 1 for the page and 6 for the images.

What else would you like to know?

[spiky]

If the page has been "cached" and the surfer clicks on the "cache" hyperlink, will this show in any stats ?

[mark_roach]

I assume you are are talking about the google cache.
If the surfer only views that page then you will be none the wiser. If they follow any of the links from your cached page then then that will show up in your logs.

[tedster]

mark and spiky,

There is a way to see google cache hits in your server logs.

Google caches only the HTML, but not the images (saving disk space, I'm sure). The images for your page are retained as links to your server. When someone clicks on the "cached results" button in Google, their servers call for the images from your server.

For each of these hits on images, instead of the simple searchphrase, you see cache:www.mydomain.com/ searchphrase.

Also, watch out for this one: each image request made by the Google cached page can come from a different IP. So if there are 7 images on a page, it looks like 7 uniques for that one click on the Google cache button. It can also look like the search phrase was used 7 times when it was only searched on once.

[mark_roach]

Tedster - Of course you are right, thanks for putting me straight. You have also highlighted another reason why the number of uniques is such a worthless figure.

[Brett_Tabke]

Spiky, no a cache hit will not show in your logs using standard tools.

I assume you are talking a browser cache or proxy cache like AOL's. Those are not tracked by standard logging utils since the page is actually never requested from the server.

In order to track such page views, you would need to use a system like Tedster describe. One possiblity is using an "IMG" call tag to your counter that is CGI in nature.

Suppose your cgi logger is at "domain.com/cgi-bin/logger.cgi". In order to over ride a browser cache or a proxy cache you could do a couple of different things:

1- call your logger with a dynamic url "img src='domain.com/cgi-bin/logger.cgi?nothing'.

That type of call would over ride most browser caches and proxy cache. They won't cache a cgi generated file. One step better is to replace "nothing" with a random string so that the browser thinks that it is truely dynamic.

2- Using a http header on the IMG that restrict caching or uses the header EXPIRES tag.

Aol will honor the expires tag and most browsers will too (not all).

Those are the only real choice available. You can try a JS script with a dynamic doc write (again, a unique string), but your results will only be marginally better than nothing.

This is the hardest thing there is to tracking on the web - getting semi accurate data. The biggest hurdle is AOL's proxy cache. It can really throw your numbers for a loop. Especially if you have good rankings on AOL's Netfind search engine.