homepage Welcome to WebmasterWorld Guest from 54.211.231.221
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Spiders
how can I make one / find one (cgi or perl)
luckynh




msg:395231
 3:09 pm on Apr 5, 2001 (gmt 0)

I have been looking for a cgi / perl spider that will spider any inputted url and follow links within the site.

I just set up MySql on my Linux Box and I would like to spider my clients sites and enter the all the gathered info (meta data,title,alt,100%text)into MySql Database.

I would like to evaluate a demo if someone can point me to the right place.

 

littleman




msg:395232
 5:09 pm on Apr 5, 2001 (gmt 0)

There is a turnkey example at [xav.com...] . It is a very good script and it is a very good value.

luckynh




msg:395233
 5:20 pm on Apr 5, 2001 (gmt 0)

Thanks littleman.

I'll check it out and let ya know

Lucky

Brett_Tabke




msg:395234
 5:21 pm on Apr 5, 2001 (gmt 0)

If you have installing perl modules figured out, download the Robots perl module. Inside is a script called "poacher" that works great.

luckynh




msg:395235
 5:34 pm on Apr 5, 2001 (gmt 0)

Thanks Brett
Do you have a link?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved