Unusual activity analyzer

[martin]

I'm looking for a log analyzer that will not report usage statistics, etc. but rather it will calculate the average and then report any unusual activity.

Like for example: I can define a treshold of 30 pages/hour, and if there are more than 30 requests for pages from a given IP/C class network the program will report that.

I'm not looking for this kind of stats, anything else that's unusual will do (user agents, etc.).

If the program is very flexible that's good too, I'm using Analog for usual stats ;-)

[mnorton]

You would really want to write something custom for that probably in Perl or something similar.

Shouldn't be too hard :?

[martin]

Nothing already available?

[mnorton]

I could write something for you that would do this if nessary all I would need is a couple of line of your log file to ensure that I have the correct format, Please bear in mind that this won't be the best script in the world.

M :)

[martin]

Mnorton, I can write it myself. I was looking for something readily available because I haven't used C (and don't really like messeing with low level languages) and I don't think a Perl script can match the performance of a well written C program.

[Hawkgirl]

Sorry this reply is so late - I'm having trouble keeping up with the volume these days.

I'm curious as to what are you going to use the data for ... would you mind sharing?

[martin]

1) I need to ban illegal crawlers, the easiest way to notice such activity is if you look for something odd like a missing or uncommon user agent, too many requests from a given IP/C class network.

2) I'd like to know is if there is downtime, I can already achieve this by using the Analog hourly report but I don't think I can do it with a 15 min report. Because such an analyzer will look for odd things it should be possible to notice shorter periods of downtime.

3) Too popular/unpopular pages at the site.

4) A sudden change in the traffic, like dropped from Google, a link from a very popular site, etc.

I'm sure you can think of a lot more things on your own.