Scooter24 - 9:33 pm on Aug 5, 2002 (gmt 0)
About, ibrowse and tivraspider were in several .htaccess files I found in the Internet - so there must be something wrong with them. There was a thread about incywincy explaning why this is a bad spider. icab can be used to make an offline copy of a site. I caught voilabot doing something not allowed on my site, so I banned it.
I also searched download.com for offline browsing software and banned everything I found.
My site has valuable content, so I ban everything which might even remotely be used to make a copy of it. I also installed several robot traps and ban the providers of users who have tried to download my site.
Before I set up all these things my site (270MB) had been downloaded three or four times in one month. Since then there have been several attempts to download the site, all unsuccessful. Just one, max. two "innocent" users/day can't access my site because of this.
I'm not paranoid, it's just that the site has valuable content. I don't mind if people make copies of individual files, but not of the ENTIRE site.