Msg#: 4521477 posted 5:49 am on Nov 24, 2012 (gmt 0)
Crawler4j is an open source Java crawler which provides a simple interface for crawling the Web. You can setup a multi-threaded web crawler in 5 minutes!
So you can download it, and use it to crawl web documents. This does nothing in itself. The data you retrieve still needs to be processed. This tool does not do that for you.
I block many terms found in UAs including: spider, crawler, scrape, download, etc. But there are some beneficial actors that may also include some of these terms, so you need to allow the ones you like.