Welcome to WebmasterWorld Guest from

Forum Moderators: coopster & jatar k & phranque

Message Too Old, No Replies

looking for search engine script

12:37 pm on Mar 19, 2008 (gmt 0)

5+ Year Member

Hi all

i wanted to have a search engine to crawl other websites in www, i heard that perl is better suited to do this as this has good text manupulation tools, CGI and it is bit faster as well....now my question is can i get any perl coded script from anywhere


1:38 pm on Mar 19, 2008 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

there are a bunch of crawlers/spiders/bots around, you'll just have to search your search engine of choice and try them out
3:38 pm on Mar 19, 2008 (gmt 0)

5+ Year Member

i tried finding them..but couldnt find them....i will be thankful to you if send me their names using stickymail...
4:27 pm on Mar 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Take a look at Perl LWP::RobotUA

The basic functionality of this class will let you create a crawler with very little code.

What you do with the info it collects is another thing.

2:09 pm on Mar 20, 2008 (gmt 0)

5+ Year Member

thanks lexi
but the problem is i am newbie to perl programming and only played with small perl programs i have downloaded the LWP class but using that class into my web pages are whole new ball game to me....
5:04 am on Mar 21, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Your first post implied you want to build a search engine -- your last post implies you just need a search box on your own website to allow your users to search the web.

What exactly are you trying to accomplish ?

8:10 am on Mar 21, 2008 (gmt 0)

5+ Year Member

if i wasnt clear enough then let me explain once again....i wanted to create search engine which may be doing task different from generally search engine do...but search engine is not to crawl my website but to index the information from other pages that i manually enter from web

I have understood my problem lately that LWP library is not installed in the web my server....are web server by default do not include these library and require personal request to install those LWP library.?

Another issue is that i am working from windows workstation and having trouble installing the library i think we need to use "MAKE" command to compile and install it which are not supported by Windows...do we have something like LWP package for windows.....?

9:18 am on Mar 21, 2008 (gmt 0)

5+ Year Member

I would like to share my recent improvement...that my web server has given me the list of perl modules which i am happy to see that complete LWP module installed

to install perl on windows i have found a good link here


which give insightfull information for unix as well as window installation..if anyone know more on this please share with me