Harvesting and caching maybe?

how can I create a cache file or something similar or the results of a cgi

[dawlish]

Essentially I'm using a cgi script to grab content from another site and insert into mine using ssi. This works reasonably well although the process is quite slow.

Is there a way in which the script could generate a server cache file or something similar that would update say once a day? This would them make the page download a lot quicker for the visitor and save on server resources as the script would only be run once a day.

Alternatively is it possible to grab content from another site and create a html file on my server? I could then used cron to run the process once a day.

I'm new to scripting and do not have a great deal of knowledge and would be grateful for any advice.

[gethan]

> Alternatively is it possible to grab content from another
> site and create a html file on my server? I could then used
> cron to run the process once a day.

Yep - totally. Take a look at wget - unix app. Can be set to grab an entire site or just individual pages.

to get a site the basic syntax is "wget -r http*//www.targetsite.com/"

You'll get the content in a www.targetsite.com directory.

Try it out and you'll quickly see how its possible to set up a mirror of content. Though if your doing this you'll need to get permission from the site or you'll get blocked very quickly. (though this may happen with what your currently doing sooner ;))

Other caching related stuff - since I'm on a roll. Hopefully someone searching in the future may find this useful.

If your actually trying to setup a caching server for an intranet look at squid - another free unix app. Can be set to do basic caching, load balancing or more.

Finally if you are trying to produce a local cache of dynamic content to save on your resources and speed up responses - apache has mod_rewrite - it can be set to check for the existance of a file, before requesting it from a cgi. The cgi could then be written in such away that it causes a file to be written as well as the content delivered. Subsequent requests for the file would then be from the "cache". A cron should be used to rm the old cache files.

I'm sure someone will reply with some windows options also.

Good luck and welcome to wmw,

Gethan

[sugarkane]

In Perl, you could use LWP:Simple to grab the data, write it to a file, and use the stat command to check the age of the cached file and update it if necesary. Eg:

[perl]
#!/usr/bin/perl
use LWP::Simple;

$max_age=86400; # max age of cached file in seconds
$filename="cache.dat"; # name of file to use as the cache

@stats=stat $filename;
if ( time - $max_age > $stats[9] ) { # check if cache needs refreshing
$foo=getstore("http://somewhere.com/content.html", "$filename");
}

# output cached data
open(FP, "$filename");
while (!eof(FP)) {
$line=<FP>;
print $line;
}
close(FP);
[/perl]

That would refresh the cache if it is more than a day old, or just output the file if not.

[dawlish]

Thanks, that is really helpful.