homepage Welcome to WebmasterWorld Guest from 23.23.8.131
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / Perl Server Side CGI Scripting
Forum Library, Charter, Moderators: coopster & jatar k & phranque

Perl Server Side CGI Scripting Forum

    
Pulling info from a SE
1goodpuppy

10+ Year Member



 
Msg#: 3350 posted 4:26 am on Nov 29, 2003 (gmt 0)

Does anyone know how to go about pulling information from a search engine to be used in a script. for example, If I wanted to pull the total number of sites that came up when I searched Widgets on HotBot?

Thanks

 

jatar_k

WebmasterWorld Administrator jatar_k us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3350 posted 9:37 pm on Nov 29, 2003 (gmt 0)

Have you looked at LWP [perldoc.com]?

Just be careful with automated queries, the Search Engines don't take too kindly to it. If you start hammering the serp's they will notice.

bobbee

10+ Year Member



 
Msg#: 3350 posted 4:37 pm on Dec 1, 2003 (gmt 0)

I haven't scripted HotBot but:

Google has an API you can access w/ scripts (limited to 1k searches per day).

I have scripted queries to AltaVista by building the query into the URL & retrieving the results w/ fopen & then using Grep to parse out the hits of intererst.

I hope this helps,

Bob

sugarkane

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3350 posted 11:45 pm on Dec 1, 2003 (gmt 0)

I second jatar's suggestion of LWP

Basically, you use LWP to pull the SERP of your choice, then parse the returned content to pull out the info you need (usually done with regular expressions, but there are also several HTML parsing modules [search.cpan.org] that might fit the bill)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Perl Server Side CGI Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved