Bot-watching

new to this concept ... how best to do this?

[jcmoon]

There's a lot I know about websites, and there's a lot I don't. Observing bots is a brand-new concept for me.

We have a few sites on dedicated servers, so we have log files with info on what bots are visiting. But it's an ocean of data, and I need a good ship to get through it.

So here's my question: how do you keep an eye on your logs? Do you retrieve the files and have some software analyze them? Do you install analysis software on your server, which you log into? Does your host provide bot- or useragent-analysis services to you?

I know this is something I *need* to do, so now I'd like to hear opinions on *how*. Apologies if this isn't the right place to ask, but seems like either this forum or the robots.txt one were best.

Your suggestions?

[volatilegx]

Personally I use home-rolled CGI scripts which keep an eye out for bot-like behavior and log it. Then I get an email each day with the results.

There are commercial bot-tracking software packages available, though, but my specialized needs aren't met by them very well.

Another good forum to get this type of info is the Tracking and Logging forum [webmasterworld.com], but the topic is appropriate here, too.

[keyplyr]

My methods, while not sophisticated, work adquately for me. A couple times a day, I download raw_logs from server, and run them through analog [analog.cx] which will display quite a lot of info depending on how you config it. Most bots will show up in the report, but some will not so I also drill through the logs manually a bit, adding new UAs to my list.

[tomda]

I am not responding to your question which specifically ask how to deal with log files but just want to tell you that a "bottrap+email notification" is great to catch bots behaving badly (not following your robot.txt).

Otherwise, I do like said above. I Download my log everyday and run it through a small PHP script which gives me the basic informations.