Is there analysis software that keeps a database or spider crawls?

It'd be great to see a summary of which pages were crawled when, by whom.

[alpine]

Or is this a function of some general log analysis packages? We use NetTracker for Windows, and it has the data, but it is not easy to see which pages Google crawled yesterday, or which they crawled for the first time yesterday, or on which days a particular page was crawled (by Google, or another spider).

Didn't find anything via a search of the WebmasterWorld archives...

[WebRankInfo]

I developed such a tool (FREE). If you're interested, sticky mail me.

[uci_bink]

I am also very interested in something like this. I think it would be very usefull to be able to tell if google was reaching all my dynamic pages.

Let me know how I can get it please!

Kevin

[Sinner_G]

If you have access to your log files, you could just go through them (using grep for example) and looking for the GoogleBot User Agent or IP adresses.

[uci_bink]

Well I do have access to my logs and have gone through using grep however having a bunch of dynamic pages (over 20,000) It is hard for me to tell from that which pages google has seen and which it has not. Im worried that because of my link structure some pages are unavailable to google.

I need some kind of program like the above poster mentioned he has...I hope he gets back to me!

Kevin

[alpine]

WebRankInfo was nice enough to offer his s/w; only problem for me is that it is for php, which we don't run. Anyone else know of anything?

[WebRankInfo]

instead of replying to all these sticky mails, I suggest for those of you who are interested to search for "google's visits" on your prefered search engine ;-)