...How to parse search engine results fast?

...with perl, lwp and fast hardware?

[bacarell]

Hi,

I'm building a metaseach engine based on data mining techniq­ues....but
this is not important...

My question is about performances of the activity of scrapin­g search
engine results from an HTML response page.

I see that some metasearch engines (Mamma, DogPile, Vivisimo­ & C.)
present top 50 results of 3-5 search engines in about 1 seco­nds.

With my perl script I am able to retrieve top 100 results of­ Google in
about 1,5 seconds, but from only one search engine!

Somebody (very much skilled in Perl) can tell me some advanc­ed
technique (parallelism, thread...bo?) to retrieve from 3-5 s­earch
engines very fast? (Hardware not included in this issue, I h­ave a fast
hardware)

Excuse me for my english (I'm italian) and for my poor Perl ­skills.

Thanks,

VB

[SeanW]

Take a look at LWP::Parallel :)

Sean