I'm finding out just how easy it is to use Perl and LWP. There seem to be all sorts of useful applications for it. If I'm not using a lot of bandwidth, and just fetching a few pages from a site each day, do I need to use code like $browser->agent(whatever) to pretend to be browser, and wait a couple seconds between fetching pages? Does anyone using spiders know what the pitfalls are, and have any general advice?
