mack - 3:26 pm on Sep 9, 2009 (gmt 0)

I agree that opt in would be a much better solution than opting out. Most people who use your application will simply have no idea what it is doing.

I don't have a lot of knowledge of distributed crawlers, but not only does the app need to spider the content, it needs to return the data to you. Does this not mean you need quite a large scale of bandwidth at your end? in theory all the combined spidered data needs to be returned to you (again using someones internet connection). If you already have large scale bandwidth available, why not spider?

Incidentally shiondev, welcome to WebmasterWorld

Bet you're glad you joined us :)

Mack.