Software to test crawl-ability

[mitchmu]

Hi,

Does anyone know of a utility that we can use to test the crawlability of a site? We would like to point this software to our site, have it crawl the site, and report back on which pages could be crawled and which couldn't. Ideally this software would "know" the rules used by google, and other spiders, and would tell us which are crawlable based on the rules for each engine.

Does such a thing exist?

Thanks,

Mitch

[bigjohnt]

You could start with the Sim Spider at Searchengineworld.
[searchengineworld.com...]

It really is amazing the tools that you find at WebmasterWorld and SEworld!

[mitchmu]

Thank you so much! That was exactly what I was looking for!