Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: phranque
I have a large site with thousands of files for people to download. (not the one in the profile) I have run into a few peopple using programs like Interanarchy to leech the whole website, by following links and saving it all.
Driving my site to sends gigs of data, in the end costing me.
Any suggestions or cgi's etc you recommend?
One is to not allow downloads unless the referer that the browser provides is from your site. Not very effective because some people don't send referers anyway, some like to use an ftp client other than a browser after they have navigated your site to the downloads area, and the bad guys can fake it. It is rather easy to implement.
Another way is to put a link on every page to a CGI script. If this script is called, it should ban that IP address from your webserver for a little while (maybe a minute or 5.). Then you make the links "unclickable" by not including any text in it. This way, a regular browser will never see the link and click on it, but a site mirroring script will see it and try to mirror it. When it does it gets banned. I'm not sure if I explained that well, but your links should look something like:
notice there's no text there to click. You might want to use a better name for the script than that.
There are other ways, but these are two that provide big "bang for buck"
When the counter reaches your max alloted downloads, lock out the IP address or send a simple text file stating that they have reached their limit within say 24 hours.
You can either make the cgi self cleaning, ie. update the db by dropping first every IP that is stale, or just run a cron job once an hour to do the same.