Welcome to WebmasterWorld Guest from 54.167.202.184

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Spiders

how can I make one / find one (cgi or perl)

     
3:09 pm on Apr 5, 2001 (gmt 0)

Full Member

10+ Year Member

joined:Jan 4, 2001
posts:222
votes: 0


I have been looking for a cgi / perl spider that will spider any inputted url and follow links within the site.

I just set up MySql on my Linux Box and I would like to spider my clients sites and enter the all the gathered info (meta data,title,alt,100%text)into MySql Database.

I would like to evaluate a demo if someone can point me to the right place.

5:09 pm on Apr 5, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


There is a turnkey example at [xav.com...] . It is a very good script and it is a very good value.
5:20 pm on Apr 5, 2001 (gmt 0)

Full Member

10+ Year Member

joined:Jan 4, 2001
posts:222
votes: 0


Thanks littleman.

I'll check it out and let ya know

Lucky

5:21 pm on Apr 5, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38074
votes: 17


If you have installing perl modules figured out, download the Robots perl module. Inside is a script called "poacher" that works great.
5:34 pm on Apr 5, 2001 (gmt 0)

Full Member

10+ Year Member

joined:Jan 4, 2001
posts:222
votes: 0


Thanks Brett
Do you have a link?
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members