rohitj - 7:01 pm on Aug 8, 2006 (gmt 0)

Hannay -- I was thinking the database load not on the crawling aspect, as that's been well-established, but rather storing and keeping these seperate indexes for each individual. To some extent, it might be like gmail with keeping a mailbox/emails but I could see this quickly going out of control because. Even if you surf 15 or 20 sites a day and add them to the sites you want, that can add up to several thousand in a year. How scalable is it to keep track of "good sites" that people want to be able to search through when each person adds a couple thouasand links? Being able to build mini search engines will take a phenomenal amount of computing in the storage and search itself. The crawling isn't so bad, as this might be done once every 3 or 4 days tops.