homepage Welcome to WebmasterWorld Guest from 54.226.93.128
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Crawler software for really big sites
Do you know any that can crawl up to 1 million pages?
Josuah




msg:4282858
 8:32 am on Mar 17, 2011 (gmt 0)

Does anyone have experience with a crawler software on big sites with around a million pages? When is not possible to do it by parts or categories.

I have tried <snip> and some other, but they can't manage all the data and the computers runs out of memory or similar.

Thanks.

[edited by: goodroi at 10:57 am (utc) on Mar 17, 2011]
[edit reason] Please no product mentions [/edit]

 

goodroi




msg:4282896
 10:59 am on Mar 17, 2011 (gmt 0)

To be honest I have not found one that had all the features I needed and ended up with a custom built one. You can search Google for product suggestions or just have a custom crawler built for you.

Josuah




msg:4285135
 4:33 pm on Mar 21, 2011 (gmt 0)

I was thinking about programming it myself but I want to be sure before spending time on it..

Thank you.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved