Welcome to WebmasterWorld Guest from 184.73.126.70

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Spiders

how can I make one / find one (cgi or perl)

   
3:09 pm on Apr 5, 2001 (gmt 0)

10+ Year Member



I have been looking for a cgi / perl spider that will spider any inputted url and follow links within the site.

I just set up MySql on my Linux Box and I would like to spider my clients sites and enter the all the gathered info (meta data,title,alt,100%text)into MySql Database.

I would like to evaluate a demo if someone can point me to the right place.

5:09 pm on Apr 5, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



There is a turnkey example at [xav.com...] . It is a very good script and it is a very good value.
5:20 pm on Apr 5, 2001 (gmt 0)

10+ Year Member



Thanks littleman.

I'll check it out and let ya know

Lucky

5:21 pm on Apr 5, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If you have installing perl modules figured out, download the Robots perl module. Inside is a script called "poacher" that works great.
5:34 pm on Apr 5, 2001 (gmt 0)

10+ Year Member



Thanks Brett
Do you have a link?