Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Crawler software for really big sites

Do you know any that can crawl up to 1 million pages?

8:32 am on Mar 17, 2011 (gmt 0)

New User

10+ Year Member

joined:Mar 27, 2009
posts: 23
votes: 0

Does anyone have experience with a crawler software on big sites with around a million pages? When is not possible to do it by parts or categories.

I have tried <snip> and some other, but they can't manage all the data and the computers runs out of memory or similar.


[edited by: goodroi at 10:57 am (utc) on Mar 17, 2011]
[edit reason] Please no product mentions [/edit]

10:59 am on Mar 17, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 380

To be honest I have not found one that had all the features I needed and ended up with a custom built one. You can search Google for product suggestions or just have a custom crawler built for you.
4:33 pm on Mar 21, 2011 (gmt 0)

New User

10+ Year Member

joined:Mar 27, 2009
votes: 0

I was thinking about programming it myself but I want to be sure before spending time on it..

Thank you.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members