moltar - 6:05 pm on Jul 14, 2005 (gmt 0)
Perl can do a lot for ya! If the job is not extremely complex, I suggest to look at wget and Perl combination. wget has a good crawler already built in. It can feed the data via pipe to a perl script. You can use the perl script just for parsing (what it's best at). Everyone is happy :). Builing a good crawler is a lot of work in any language.