brotherhood_of_LAN - 7:40 am on Sep 9, 2009 (gmt 0)

Interesting thoughts, I wouldn't be quick in condemning 'the ability to spider the web through a distributed crawling network', but there are some important issues that have already been brought up in the thread.

The ability to verify real/mock UA's would appear to be fairly important.

Regarding bandwidth issues, it would make sense and be transparent for the installer/software to provide the user with an option to cap how much bandwidth is dedicated to the spidering cause.