traffic volume?

[Dan99]

Pardon the newbie question, but I'd like to understand if Apache keeps track of the byte-volume of served traffic. I'm considering a commercial server that has traffic volume caps, and I'd like to know how close I am to those caps. Does the weekly log have such a number somewhere? How do you establish traffic volume for material served?

[TorontoBoy]

My host provider has limits on physical and virtual memory. These are displayed in nice chart form by hour of the day. If I exceed the physical memory my server will use cache memory and then return 504s (server busy) to people. I get a bunch of usage stats under "CPU and Concurrent Connection Usage". I also have I/O Usage and Server faults.

I have no bandwidth limits that I remember, but then again I kill bots and minimize my bandwidth usage. They have never complained to me, but past ISPs have temporarily shut down sites due to excessive CPU usage. Prior to bot killing I had a huge problem with excessive use of bandwidth.

AWStats has bandwidth used numbers, which might help you, broken down by hour, country, IP, etc. Note that rogue bots can really kill your bandwidth. AWstats is not very specific for identifying bots.

In choosing your host provider check for physical and virtual memory limitations as well. How much memory will they give you?

[Dan99]

Yeah, I kill bots routinely, and it's an important strategy. But I currently have an academic host provider, and they don't give a c**p what my traffic is. I'm moving to a commercial service, and if I'm serving a lot, you bet they'll make me pay for it. My recollection is that Apache keeps that number somewhere, or provides a tool to look it up, but I don't recall where. My target host provider has no memory or bandwidth restrictions.

[whitespace]

What format does this "weekly log" take?

The standard Apache access log does include the bytes sent in the response body (but normally excludes the response headers). The access log can be customised to include the total data sent and received (including headers), but you need server access to set this. Using a log file analyzer (or stats tool) you can easily total this figure. (Or you can write your own script, if you want the exercise? The access log is a plain text file.)

[Dan99]

Sorry. I do access logs weekly. I have server access. How do I set it up to report the total data sent? If that's not possible, yes, I could do a script to add it all up.

[lucy24]

If you're currently on some form of shared hosting, they may well have the information already. You'll need to go digging through the deeper depths of your control panel or equivalent, possibly allowing for a 24-hour delay while they do the calculation. I know that yoincks ago, my host used to charge for bandwidth. So the information is still collected, even though they no longer do anything with it.

If you don't yet have access to the server--meaning that you can't change the logging details--it's still pretty straightforward. The ordinary access logs give you the bytesize of each file sent out, exclusive of headers. Now find out which headers are sent out with each type of file, work out the size, and then multiply by the number of responses.

Double-check the server's billing criteria, though. They might up and decide to charge for some other element, like CPU usage instead of (or in addition to) bandwidth.

[Dan99]

I'm not on shared hosting. I have a Mac running Apache. It's my machine. I'm not billed. No one else even has an account on it. No one else looks at the logs. I have all root privileges. But seriously, I could have sworn there was some Apache tool to get that traffic number. Yeah, I could do up a script to add up all the numbers in the log, but if Apache has already done that for me somewhere, I'd like to know.

[lucy24]

:: wandering off to research, as this isn't something I can investigate personally ::

Oh. OK. You want mod_logio, which works on top of mod_log_config. Predictably 2.4 [httpd.apache.org] adds a couple of options that weren't available in 2.2, but essentially it's the same mod with the same functionality. Note that this logs request/response size including headers; that's what makes it different from the number already visible in your ordinary access logs.

[Dan99]

Hmmm. Doesn't that just enable logging size per request? I already have that. But what I want is the sum of all those log sizes over a specified time period. Again, I suppose I can just make a script that adds up all the response size bytes in my log file. But I was hoping there was some utility that would do that for me. I suppose I could just get a log analyzer for a free trial period, and use that.

[phranque]

as TorontoBoy suggested awstats will provide the data you seek.

[Dan99]

Sourceforge is down.

[robzilla]

You could try GoAccess. Install it with Homebrew, point it to your logs and it will give you real-time insight into your bandwidth usage (and many other things), or you can have it generate an HTML report. Alternatively, there are lots of log analyzers for Windows and you could use Wine to run them on your Mac (or find a Windows computer and run the analysis there).

Munin will probably install on a Mac, too, and may give you a better picture since Apache does not log all network traffic, as whitespace noted. Of course, if you also use this Mac for other purposes, your own bandwidth usage will be included.