Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Best way to track search engine crawler

6:36 am on Jul 13, 2014 (gmt 0)

New User

10+ Year Member

joined:July 8, 2006
votes: 0

Tools like google analytics dont work because they track visa javascript which most of the crawler wont run.

So the best way will be to parse the server logs I guess. But what tool can we use to parse and display the trends and stats for bots and crawlers?
7:00 am on July 13, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
votes: 316

Does your server have stats programs? Since I don't know your hosting environment, I don't want to guess which might be available. The most common Apache servers give you choices via the control panel.
7:42 am on July 13, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
votes: 832

The default analog stats package that most hosts offer is pretty useless. One non-Google analytics program has an alternative version that works by analyzing your site logs. But you may find it just as useful to roll your own, using the language of your choice. The vast majority of robots have distinctive behaviors that are very different from humans, so you can sort them out easily. For many sites it's also useful to separate your human visitors: page visits on one side, image alone on the other.

If you're making up your own stats program, it's easy to add and subtract things you want to track, like the botnet du jour or requests for specific files.