I've used Spidersoft WebZip for this for a long time to great effect.
bull
6:57 am on Aug 2, 2004 (gmt 0)
Will this UA show as SpiderZilla in access logs?
Spiderzilla is a UI for HTTrack 3.0. It will take over the Browser's UA and NOT look for robots.txt It will make HEAD requests on filetypes that are not html, htm, gif, ... (all the web stuff).
amznVibe
7:02 am on Aug 2, 2004 (gmt 0)
Be aware if this is not for your own site or client, you will make another webmaster very upset.