Forum Moderators: coopster & phranque

Message Too Old, No Replies

Searching in google

I'm trying to search in google from my CGI script

         

mrfori

3:24 am on Jul 16, 2003 (gmt 0)

10+ Year Member



I am trying to search in google and extract the results .. but the documents comes back blank, here is a fragment of my code

use LWP::UserAgent;
use HTML::LinkExtor;
use URI::URL;

$url = "http://www.google.com/search?hl=en&lr=&ie=ISO-8859-1&c2coff=1&q=school";
$ua = LWP::UserAgent->new;

.
.
.

$request = HTTP::Request->new('GET', "$url");
$res = $ua->request($request, sub {$p->parse($_[0])});
my $base = $res->base;
@imgs = map { $_ = url($_, $base)->abs; } @imgs;

# Print them out
print join("\n", @imgs), "\n";

what am i doing wrong? the code work if I use yahoo
$url = [search.yahoo.com...]

sugarkane

9:11 am on Jul 16, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to WebmasterWorld mrfori

I've not checked your code, but as Google really, REALLY don't like automated queries, it's likely that the standard LWP user agent is blocked. Have you tried setting a different user agent?

If you want to search Google in a way that complies with their TOS, you could try looking at the Google API [webmasterworld.com]

bilalak

10:18 am on Jul 16, 2003 (gmt 0)

10+ Year Member



Try to send a forged Agent. or as they said use API
for 1000 queries daily only. I think the forged agent has some limitations also.

The best solution is to use lynx on nix and then manipulate the output