Welcome to WebmasterWorld Guest from 54.147.10.12

Forum Moderators: coopster & jatar k & phranque

Message Too Old, No Replies

Pulling info from a SE

     
4:26 am on Nov 29, 2003 (gmt 0)

New User

10+ Year Member

joined:Aug 26, 2003
posts:17
votes: 0


Does anyone know how to go about pulling information from a search engine to be used in a script. for example, If I wanted to pull the total number of sites that came up when I searched Widgets on HotBot?

Thanks

9:37 pm on Nov 29, 2003 (gmt 0)

Administrator

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 24, 2001
posts:15755
votes: 0


Have you looked at LWP [perldoc.com]?

Just be careful with automated queries, the Search Engines don't take too kindly to it. If you start hammering the serp's they will notice.

4:37 pm on Dec 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Nov 19, 2003
posts:12
votes: 0


I haven't scripted HotBot but:

Google has an API you can access w/ scripts (limited to 1k searches per day).

I have scripted queries to AltaVista by building the query into the URL & retrieving the results w/ fopen & then using Grep to parse out the hits of intererst.

I hope this helps,

Bob

11:45 pm on Dec 1, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 6, 2000
posts:904
votes: 0


I second jatar's suggestion of LWP

Basically, you use LWP to pull the SERP of your choice, then parse the returned content to pull out the info you need (usually done with regular expressions, but there are also several HTML parsing modules [search.cpan.org] that might fit the bill)