Welcome to WebmasterWorld Guest from 54.226.241.8

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Spiders

how can I make one / find one (cgi or perl)

     

luckynh

3:09 pm on Apr 5, 2001 (gmt 0)

10+ Year Member



I have been looking for a cgi / perl spider that will spider any inputted url and follow links within the site.

I just set up MySql on my Linux Box and I would like to spider my clients sites and enter the all the gathered info (meta data,title,alt,100%text)into MySql Database.

I would like to evaluate a demo if someone can point me to the right place.

littleman

5:09 pm on Apr 5, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



There is a turnkey example at [xav.com...] . It is a very good script and it is a very good value.

luckynh

5:20 pm on Apr 5, 2001 (gmt 0)

10+ Year Member



Thanks littleman.

I'll check it out and let ya know

Lucky

Brett_Tabke

5:21 pm on Apr 5, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If you have installing perl modules figured out, download the Robots perl module. Inside is a script called "poacher" that works great.

luckynh

5:34 pm on Apr 5, 2001 (gmt 0)

10+ Year Member



Thanks Brett
Do you have a link?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month