| Crawler software for really big sites
Do you know any that can crawl up to 1 million pages? |
Josuah
msg:4282858 | 8:32 am on Mar 17, 2011 (gmt 0) |
Does anyone have experience with a crawler software on big sites with around a million pages? When is not possible to do it by parts or categories.
I have tried <snip> and some other, but they can't manage all the data and the computers runs out of memory or similar.
Thanks. [edited by: goodroi at 10:57 am (utc) on Mar 17, 2011]
[edit reason] Please no product mentions [/edit]
|
goodroi
msg:4282896 | 10:59 am on Mar 17, 2011 (gmt 0) |
To be honest I have not found one that had all the features I needed and ended up with a custom built one. You can search Google for product suggestions or just have a custom crawler built for you.
|
Josuah
msg:4285135 | 4:33 pm on Mar 21, 2011 (gmt 0) |
I was thinking about programming it myself but I want to be sure before spending time on it..
Thank you.
|
|
|
