Forum Moderators: Robert Charlton & goodroi
The ones that seems to crawl my entire site run for days and never complete, the other complete but only provide a very high level sitemap, they don't crawl all of the way into my site, they produce a xml file with a couple of hundred lines, very high level.
One of the tools that seems to be popular with people (GSite&^*$&) I left to run for 2 days, it crawled 40,000 pages and says its about 50% complete, the performance get gradually worse.
What do people use on very very large sites?
Thank you