Welcome to WebmasterWorld Guest from 18.104.22.168
This website offers an exhaustive number of links regarding spider development (including some free spider implmentations).
I'd recommend that you consider a PHP solution if this is one of your first projects - you'll find tons of helpful documentation at:
The first talks about building spiders and parsing pages. The second really gets into the specifics of getting information out of pages.
Others have mentioned using PHP and LWP. I really recommend WWW::Mechanize. It's based on LWP, so you can always revert back to it, but it adds a bunch of things to easily follow links and submit forms.