spider analysis software

[gdawg]

Wondering if anyone knew of any spider analysis software? Am looking for something that breaks out the visits of each spider by date, tells me which files they crawled during that visit, etc.. It would be awesome if it could show you a path flow report for spiders.

I know alot of logfile analyis software give you some data, but I am looking for something that has a lot of data specifically related to search engine spiders and how they interact with your site.

[wilderness]

I seem to recall that somebody offers a custom tool for this, however the reference escapes me.

Don

[Megaclinium]

I started doing this kind of stuff. Nothing sophisticated.
I read the log file records, parse them into fields.
Look up by UA text if spider advertising itself as spider & assign a name so all IP addresses that spider comes in on can be sorted together.
Add a few more spiders by address if nothing in UA.
create a flag 'bot' = true if spider value not null.
write this out to a temp database table.
sorting several ways, summing in some of them.

this lets me sort out actual hits vs bot hits,
see who is creating errors by IP address, bot name, etc.
I was curious about the home brew spider in previous post so I wrote a quick prog to extract all that address and see what hours of the day it was running. Showed only consecutive hours running on consecutive days. So someone might have been running it only at work.

Lots more you could do.
I noticed some bots from dift addresses because of patterns were actually the same bot.