Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Need software to build 8m page custom index

ISO large scale index/crawler, recommendations?

2:22 pm on Sep 26, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 11, 2004
votes: 0

I have looked around at vaious software that can help me transform my metasearch engine into a custom built searchable index of 8 to 10 million pages.

The metasearch option is delivery so-so results and hope this upgrade will deliver supreme style results.

Unfortunately, haven't found one that can handle what I need to do. All too small or require too much hardware.

If anyone can recommend a pre-made sw package that can crawl/index of 8 million pages and deliver a customized search results, please contact me as soon as possible. I know the SW exists, just can't get a grip on what can do the job as need. I checked with siets.net, they came close, but too much hardware requirements were needed.

---Added, how hard/how much would it cost to pay a programmer to build a this program that can do such a thing? Where would I look to find one? Is it possible to for a perl man?

My whole goal is to improve the serps beyond what a metasearch engine can do. I want to improve serp results for my users - bottom line necessity if I am going to dominate my competitors.

Thanks you so much for your help,

Kind Regards,


7:36 am on Sept 28, 2005 (gmt 0)

New User

joined:Feb 2, 2005
votes: 0

Looking for SE/crawlers/indexers in Database forum will not get you many results (-;