Welcome to WebmasterWorld Guest from 23.20.75.214

Forum Moderators: coopster & jatar k & phranque

Message Too Old, No Replies

Grab source code from any supplied URL

There must be an easy way to do this...

     
9:51 pm on Dec 3, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 4, 2003
posts:64
votes: 0


Hi, I'm would like to have a form with a field where the user supplies any URL. Clicking submit then grabs the source of the page specified in the URL.

Does anyone know the basic syntax for this? I'm gathering one uses the CGI module...

Thanks,
skube

10:29 pm on Dec 3, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member drdoc is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 15, 2002
posts:6807
votes: 0


You don't have to use the CGI module. Have you looked into LWP?
11:28 pm on Dec 3, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 4, 2003
posts:64
votes: 0


Ah right, thanks Dr. Doc...so something like this?

use CGI;
use LWP::Simple;
$query = new CGI;

$webpage = $query->param("web_url");
my $source = get $webpage ;

11:36 pm on Dec 3, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 4, 2003
posts:64
votes: 0


But now I have another problem, I have the source of the URL stored in the $source. However, I wish to convert each source line into an array @source, to do further manipulations.

Usually, one would open a file and assign each line using <> as below:

open(DAT, $data_file) ¦¦ die("Could not open file!");
@raw_data=<DAT>;
close(DAT);

But for my case, I don't have a file which I'm opening, just a very long variable $source. So, I want to do something like:

@source = <$source>

But obviously, that is incorrect syntax.

12:25 am on Dec 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 6, 2000
posts:904
votes: 0


Probably best to use the split function :-

@source=split(/\n/, $source);

That will give an array with one line of code per entry.

(NB that should work for pages stored in *nix format. For Windows files you might have to use a more complicated regex to deal with the \r carriage returns as well as the \n newlines...)

4:09 pm on Dec 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 20, 2003
posts:175
votes: 0


Why not use a script? Try http,//willmaster.com/master/snooper/

You will need the LWP::Simple mod, and probably already have it installed.