Internet Archive using Nutch

[keyplyr]

***.***.***.* - - [27/Sep/2005:15:33:19 -0700] "GET /robots.txt HTTP/1.0" 200 2016 "-" "InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; [e-mail])"

IP does belong to Internet Archive. Anyone know why they're using Nutch?

[volatilegx]

Let's take this discussion outside the Search Engine Spider Identification forum. The ethics of archiving publicly accessible material (as interesting as I find it) are way outside this forum's topic.

[wilderness]

Removed per Dan's request.

Edited by wilderness.