Self-made Web Statistics on my PHP website?

If my host doesn't support real-time statistics.

[picophd]

Greetings,

Can I have real-time web statistics throughout my website by using PHP?

My problem is that my website hosting service does not offer real-time web statistics, only lousy monthly reports. Yes, I know I should change hosts and all, but until my current annual contract ends and I can afford to change my host, can I develop my website pages somehow, using PHP for instance, so they allow me to study real-time web statistics?

If yes, then how can I do this? Is there anywhere I can download ready-made PHP scripts, specifically for this purpose, to be added to my web pages after customization, for instance? And how can I, later on, study those web statistics? If any links or so may look like advertising in the reply, I don't mind to receive sticky mail at all.

I'll be very grateful for any tips and information. Thanks to anyone taking the time to help.

[lokjah]

hey pico, I use a really cool stat prog by shaun inman

"shortstat" very easy to install....

[jatar_k]

are you just looking to get stats or do you need logging too?

If you have access to logs there are tons of programs around

[picophd]

*teethy shy smile*

I'm a total newbie to the benefits of "access to raw log files".

I'm not even sure if these files that I see in a certain folder in my server are those called "raw log files", the access to which is apparently something nice. But yes, when I login into my website's server using FTP, there's a path like this:

Log ==> W3SVC75 ==> ex041112.log (to) ex041125.log

Are these what you call raw log files? I can definitely see that their contents have something with do my best site's hits and so, but I don't see "referrers" for instance. However, if these are the log files, then I most definitely am interested in how to properly read them and extract those cool graphs from them and so. Can I get referrers from them, too, with search engines and keywords used, etc?

[picophd]

lokjah, got it, mate. Thanks a lot!

I read the readme and all. However, it seems that I need a MySQL (or some database) on my server? Is this true? It's a long story "how come", but I'm not sure if I have any MySQL databases in my hosting package, still waiting for the answer from the admin. Everything is manual there, unfortunately. But I wanna be sure if I have to worry about access to a database in order to use ShortStat on my site.

[tomda]

Log ==> W3SVC75 ==> ex041112.log (to) ex041125.log

Indeed, these a logfile (one per day) so the ex041112.log is the one of 12th Novembre 2004, etc...

They are plenty of apps to read logfiles. Google for "log analyser" or "analog". If you are looking for something free which do not require asp/php/perl script, then analog is very popular.

Otherwise, it is easy to make your own log analyser with php/asp/perl and mysql.

[picophd]

tomda, thanks a bunch. I've actually been wondering for quite a while now how I can read those log files!

I got Analog, and managed to install it and configure it, everything seems to be in order except two things that bothered me, and they happen to be the most important to me actually:

1. "unresolved numerical addresses", all of them are like that. I *did* read whatever help docs from Analog that I could find about DNS, and I added the [

   DNSFILE dnscache (and) DNS write

   ] part in the analog.cfg file, but--ever since--it keeps giving me errors, not just the first time I run Analog with the DNS commands, but every time, and even when I do delete the

   dnslock

   file and run again. This basically renders the "Domain" and "Organization" reports totally useless to me, and these happen to be the most important things. Do you know what I can do to actually see *names* or at least *domain names* in the Domain and Organization reports? Can that ip-to-country.csv file from somewhere on the web be used with Analog somehow, for instance? Or how can I properly set up something to handle all the IPs in the log file properly using DNS? Maybe this requires that Analog would be somehow on my server itself, not just analyzing my downloaded log files?

2. No referrers whatsoever (with keywords used to find site, etc). I'm guessing this is because they aren't in the log files themselves? How would the hosting admin be able to send me monthly reports with "referrers" in the them, though, if those referrers aren't listed in the log files already?

[jatar_k]

sounds like you might not have extended log format

do your month end reports from your host have that info?

[picophd]

I haven't received one, yet, but when I fussed about web statistics to the admin, he said that I do get, at the end of the month, referrers, yes...my question did include keywords used to find my site and from which search engines, etc. Sorry if it seems I'm a bit clueless, just the type of hosting service I got stuck with after an irrelevant long story.

But what do you think from what I said? Maybe the admin didn't really understand what I needed, misunderstood me or something, and said yes about something else? Do you think if the raw log files don't have that info, then it's unlikely they can send it to me in the monthly report?

[jatar_k]

>> Sorry if it seems I'm a bit clueless

no apologies necessary here picophd

here is what a logline looks like from my system

129.xx.xx.xx [23/Nov/2004:23:58:41 -0800] "GET /path/page.php HTTP/1.1" 200 14490 "http://www.google.com/url?q=http://www.example.com/path/page.php" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

broken down
IP datetime request status bytes_sent referer user-agent

this might help as well
[httpd.apache.org...]

take a look at the logs that you mentioned and see if that data is in there.

[picophd]

no, jatar_k, the referrer is not there. I even made sure, right before downloading today's single log file, I made a search specifically for my site on G, and I clicked on the link from G results. And right before that, I validated my page even, just so I can see some other address that I can recognize. And in the log file, I could see tons of saliva from Slurp :) and hordes of this pattern:

2004-11-25 11:26:04 85.xx.xx.xx- 111.111.111.111 80 GET /somepic.png - 200 Mozilla/5.0+
(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.7.5)+Gecko/20041103+Firefox/1.0RC2

I just changed the IP (mine) and image name, that's all. Plus the validator. And that's it. No referrers whatsoever, no mention of G, no mention of keywords or anything.

[edited by: jatar_k at 7:04 pm (utc) on Nov. 25, 2004]
[edit reason] obfuscated ip [/edit]

[jatar_k]

well if there was no referer found for my logline it would look like this

129.xx.xx.xx [23/Nov/2004:23:58:41 -0800] "GET /path/page.php HTTP/1.1" 200 14490 - "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

just a hyphen where the referer should be, referers are not an exact science because they are not always there, regardless of whether there was one or not.

There are 2 hyphens in your line, with out being able to see the CustomLog setup one of those could possibly be where the referer is supposed to be located.

[picophd]

Interesting news to me.

I thought that if someone used a keyword to find my site, it should almost always be there, in the log files or stats, I mean. The problem is that I've done this more than once, and on different days, and for all the days that I've downloaded log files, Analog didn't show any keywords or so.

For instance, I have a free counter on a personal blog in G's blog service, and I see a lot of keywords and referrers when I log into my account at the site that gave me the free counter, and I get to know where they came from or how they found me and all. It feels really good to know this kind of info, at least for me. So it's annoying me that this is not working the same way with that other startup pro site.

[jatar_k]

I would first talk to the host and confirm whether referers are available in your logs or not and where exactly they would be

As far as referers in general, you will definitely get this data if it is available through your logs. Thing is CustomLog format is set by each individual server admin or host and sometimes the value of referer data needs to be explained to them, just be nice. :)

There is software that blocks referer data from being logged. Even in your hit counter, if you looked at logs against the counter data, I bet it would be the same. You would also find that there is not referer data as often as we would like. I am happy with 85% and up though 100% would be nice. ;)

[picophd]

Notes taken, jatar. Thank you a million really. :)

And thanks to everyone else, I've found two great tools from the folks here.