Is there a tool to monitor traffic on Apache servers?

top, apachetop, iftop, and tail don't cut it

[MichaelBluejay]

I have a dedicated server with a couple dozen domains. I'd like a tool that lets me monitor the traffic in realtime so I can see if I'm getting DOS'd/DDOS'd, or if a bot is going crazy with page requests, or where there are opportunities to optimize my pages based on how frequently they're accessed. (I guess I could do the last bit with non-realtime logfile analysis, but looking at things in realtime might give me some ideas.)

I can run "tail -f" on a specific logfile to see the requests coming in, but that doesn't let me monitor all 20-odd domains at once. It also doesn't summarize the data.

What I'm envisioning would have output something like this:

Domains 
domain1.com: x file requests/min y page requests/min 
domain2.com: x file requests/min y page requests/min 
domain3.com: x file requests/min y page requests/min 
  
Pages requested in the last minute: 
page1.html: X requests 
page2.html: Y requests 
page3.html: Z requests 
  
Most active IP's in the last minute: 
200.10.200.10: x files / y pages on [which domain(s)] 
100.20.100.20: y files / y pages on [which domain(s)] 
300.50.300.50: z files / y pages on [which domain(s)]

I looked long and hard on Google for such a tool but couldn't find anything. It seems like something like this should exist, because certainly other people have wanted that functionality besides me, and it shouldn't be that hard to program -- the program just needs to know which logfiles to watch and then parse all the data. In fact, if such a thing *doesn't* exist then I smell a market opportunity. :)

So, is there something like this already?

[spinnercee]

Two apache options will give you realtime access to the connections the server is servicing:

# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

this needs to be set/enabled first, then find this part:

# Allow server status reports, with the URL of [servername...]
# Change the ".your_domain.com" to match your domain to enable.
#
<Location /server-status>
SetHandler server-status
Order deny,allow
# Deny from all
# Allow from .your_domain.com
</Location>

The way it is above, anyone can see the stats.

You basically get a realtime connection status report for the server, showing stats for each slot including the request itself. I can't display them here, but the key looks like this.

Srv - Child Server number - generation
PID - OS process ID
Acc - Number of accesses this connection / this child / this slot
M - Mode of operation
SS - Seconds since beginning of most recent request
Req - Milliseconds required to process most recent request
Conn - Kilobytes transferred this connection
Child - Megabytes transferred this child
Slot - Total megabytes transferred this slot

This would let you see if you're getting a million hits at the same time for a single file or from a single IP -- There is also a global display of requests/sec and bytes/request, and it's also useful for helping you configure the server's max connections [ThreadsPerChild=].

Current Time: Wednesday, 17-Jan-2007 21:21:41 Eastern Standard Time
Restart Time: Tuesday, 16-Jan-2007 18:25:34 Eastern Standard Time
Parent Server Generation: 1
Server uptime: 1 day 2 hours 56 minutes 7 seconds
Total accesses: 212293 - Total Traffic: 2.7 GB
2.19 requests/sec - 28.9 kB/second - 13.2 kB/request
1 requests currently being processed, 127 idle servers

______________W_________________________________________________
________________________________________________________________
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process

It's basically a small page of plain text info -- so don't expect any fancy graphs or anything.

* I can't believe anyone runs an Apache HTTPd without using this.

[MichaelBluejay]

Thanks, but this doesn't come close to doing what I'm asking about.

[Matt Probert]

Sounds like you need some program which will constantly (perhaps pausing for n seconds) read the tail of the access logs for each domain, process the data, and send it to a file.

Maybe a Perl script?

Matt

[MichaelBluejay]

Yes, that's what I'm thinking of. But I figured that something like this *has* to already exist, right? I'm a slow programmer, and it would take me quite a while to put something like that together, and hiring a programmer would likely be pricier than buying something off the shelf. On the other hand, if a tool like this really doesn't exist, it's a market opportunity, and after hiring a programmer to write the code I could sell it to others for a gazillion dollars.

I dunno, though....it seems like a tool like this should be immensely popular, but if that were true, it also seems like someone would have created one before now.

[jtara]

Three things to look at:

(1) SNMP, and SNMP-monitoring tools.

SNMP is a protocol for gathering remote management information. Most OS's and most server applications (e.g. Apache) have SNMP built-in or through modules or plugins. In addition, many hardware devices (routers, etc.) support SNMP.

Although SNMP can be used both for monitoring and control, it is most commonly used for monitoring.

Once your OS and server(s) are set-up to use SNMP, you can use any of a huge variety of SNMP monitoring apps to poll them and display the data as you wish.

As an example of the type of information available remotely via SNMP would be CPU load, page swaps, disk usage, disk transfer rate, bandwidth, memory usage, etc. etc. etc. A webserver might report things such as open connections, bytes/second, number of unique users, number of logged-in users, etc. etc.

(2) Other management and reporting protocols. WBEM (Web-Based Enterprise Management), CIM, etc. These are less popular in the Linux world but more popular in the Windows world.

(3) Tools that integrate information from multiple sources. Nagios would be a good example of this. Nagios is able to use SNMP, WBEM, CIM, as well as simpler tools such as ping, local shell scripts, remote shell scripts, etc. It allows you to design displays that integrate the information you want in the way you want to see it. Although it has a web GUI for display, it does not have a GUI for setup. (You configure it using text files.) So, setup is a bit painful, but you can get just what you want, and it has a HUGE variety of ways you can get at the information that you want to display and log. Nagios has it's own client daemon that you can install on your servers, as well.

[MichaelBluejay]

Okay, but I'm not really interested in protocols, I'm interested in an actual tool that does something similar to what I described. Do you know of any?

[jtara]

Okay, but I'm not really interested in protocols, I'm interested in an actual tool that does something similar to what I described. Do you know of any?

Nagios.

[MichaelBluejay]

I checked out the Nagios website and it doesn't look like it comes anywhere close to providing the kind of info I listed in my original post. If you know something I don't, can't you point me to exactly where on their site I can see that their product does what I'm asking about?

[spinnercee]

Maybe you don't have a clear idea of how Apache "Server-Status" and the logs work.

You're looking to find out what's happening "right now" -- you'll never find that in any of the Apache logs -- this is because requests are not logged until after they have been serviced, so if your HTTPd is being DOS'ed you won't see that reflected in the logs until after the attack has completed -- by then it's way too late, and you'd be hard pressed to look back into the logs to see the attack reflected.

Now, the /server-status works differently, in addition to showing the slots as they are being accessed servicing requests, it shows the connection state of each request in real time -- this is important because requests that have been serviced but are still in the KeepAlive state will still be unavailable to service a new request (from a new IP) -- download accelerators can be configured to open 20 or more connections to an HTTPd at the same time for each file they want, so it's real easy for a single IP to consume 100 slots for five files -- you may not see this in the log because the requests will not likely finish or start at the same time.

The part of the report I can not show was the detail listing -- this shows IPs and the time (in seconds) that the request has been serviced among other useful things like the actual GET/HEAD/POST, etc.. request itself.

* Don't dismiss it until you've seen it -- this is what you need.

[jtara]

I think you want somebody to hand you a solution on a silver platter. If you want custom reports showing exactly what you want to see, you are going to have to do some work, no matter what package you use.

Nagios can do this, as can a number of other solutions.

It can gather information from SNMP, wbem, or even any shell script that you can dream up. (Nagios will run the shell script periodically, remotely.)

SNMP (using an Apache SNMP module, and/or SNMP reporting built-in to Linux networking) will provide the kind of traffic information you are asking for, and Nagios can consolidate this from multiple sites. But you don't want to hear about SNMP. It's a protocol. Don't want to know anything about protocols...

You could certainly write some shell scripts to "tail the logs" as you suggested, and feed that information back to Nagios. But you don't want to be bothered.

[jtara]

You might also want to check out a book called "Automating UNIX and Linux Administration". It will give you some ideas on building automated remote monitoring and administration of multiple sites. Chapter 10 is on System Monitoring. BTW, this is where I first learned of Nagios.

The key here is "building". You have to do some work. Build it step by step, and perhaps the task won't seem so daunting. It's unlikely you are going to find exactly what YOU want, right out of the box.