Google Like crawler

Looking for software to test crawl my website.

[iamjoe]

Anyone know of any software i can setup on a unix machine which will crawl my entire website and report back the number of pages/keywords and whatnot across the entire site?

Basically i want to run a dry-deep-crawl against my site with some sort of google like engine to see how it would hold up.

I know there are some crawling software on Sourceforge and some packages in the FreeBSD ports but i figured someone here would know what to use to get the best results.

Joe

[Brett_Tabke]

Start with some of the simple link checker scripts to find the 404's. A bit more involved is something like XAV's outstanding fsde engine that can crawl an entire site. After that, hit Avi's outstanding SearchTools.com and dig to find something that suits.
http://www.searchtools.com/tools/tools.html