Tracking search engine spiders and bots

[fsa3]

How do you monitor which search engines have traversed your site (and how often)?

[TonyHodge]

We use NetTracker to track referrers and keywords but they also provide robots/spiders/crawlers so you can track them. These numbers are separated from the standard metrics.

[sem4u]

Tony - you sure are a big fan of that software! ;)

[Reflect]

I have used FastStats in the past for this. It's also under $100 US.

Webtrends is also VERY nice buty a little more cash.

Take care,

Brian

[fsa3]

Nothing free out there just to get my feet wet. I also can do my own programming but I'm not sure if its worth it.

[mymotech]

Net tracker lite is a free version of Net Tracker. I imagine that it would have a spider/bot report.

You can check it out at [nettrackerlite.com...]

[edited by: jatar_k at 5:11 am (utc) on July 30, 2005]
[edit reason] linked it up [/edit]

[casimir1234]

WebTrends with many many filters being applied

[waziwazo]

I use WebLog Expert with manualy edited SearchEngines.cfg and Spiders.cfg

[bill]

I second the WebLog Expert recommendation. They have an extremely flexible set of configuration files that you can customize very easily.