Does anyone have experience with a crawler software on big sites with around a million pages? When is not possible to do it by parts or categories.
I have tried <snip> and some other, but they can't manage all the data and the computers runs out of memory or similar.
[edited by: goodroi at 10:57 am (utc) on Mar 17, 2011]
[edit reason] Please no product mentions [/edit]