Script to write file

[toolman]

Is there a script that will crawl a directory and then write the url's to a .txt file?

[Brett_Tabke]

So bascially, you are looking for a file directory reader that dumps the paths to a text file? I have something like that some where...

[sugarkane]

Or do you mean a directory as in ODP?

[toolman]

Yes BT. Writes the urls to a file.

[Xoc]

Apache or IIS?

[toolman]

Apache. What I'm after is getting a text file of urls to every page on a domain so I can feed that into a batch submission routine.

[sugarkane]

I use:

#!/usr/bin/perl
use File::Find;

$domain="http://www.foobar.com";
$root_path="/path/to/domain/root/";

find \&foo,$root_path;

open (FP,">output.txt");
foreach $i (@files) {
print FP "$domain/$i\n";
}
close(FP);

exit;

sub foo {
my $foo=$File::Find::name;
if ( $foo=~/html$/) {
$foo=~s/$root_path//;
push @files,$foo;
}
}

(Note - this crawls down the directory tree as well)

[toolman]

Thanks SK...it's not returning every file in a directory though????

[sugarkane]

It should return all html files - you can either hack the regex below or remove the whole 'if' statement if you want it to include other file types.

if ($foo=~/html$/) {
$foo=~s/$root_path//;
push @files,$foo;
}

[toolman]

Oh I see now. How would I add the extension .HTML (all caps) in there too. I have a bunch of machine generated pages and they are all HTML and not html.

[sugarkane]

Change if ($foo=~/html$/) to if ($foo=~/html$/i)

If you wanted to include, say, .htm files as well you could change it to

if ($foo=~/html$¦htm$/i)

[toolman]

Thanks SK and BT...I got it working now. Over 300 urls in one site. We wouldn't want to do that by hand would we?