Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Crawler software for really big sites
goodroi - 10:59 am on Mar 17, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4282856.htm
To be honest I have not found one that had all the features I needed and ended up with a custom built one. You can search Google for product suggestions or just have a custom crawler built for you.
Brought to you by WebmasterWorld: http://www.webmasterworld.com