Best way to track search engine crawler

[thok]

Tools like google analytics dont work because they track visa javascript which most of the crawler wont run.

So the best way will be to parse the server logs I guess. But what tool can we use to parse and display the trends and stats for bots and crawlers?

[not2easy]

Does your server have stats programs? Since I don't know your hosting environment, I don't want to guess which might be available. The most common Apache servers give you choices via the control panel.

[lucy24]

The default analog stats package that most hosts offer is pretty useless. One non-Google analytics program has an alternative version that works by analyzing your site logs. But you may find it just as useful to roll your own, using the language of your choice. The vast majority of robots have distinctive behaviors that are very different from humans, so you can sort them out easily. For many sites it's also useful to separate your human visitors: page visits on one side, image alone on the other.

If you're making up your own stats program, it's easy to add and subtract things you want to track, like the botnet du jour or requests for specific files.