Forum Moderators: phranque

Message Too Old, No Replies

How to stop scrapers?

Having a hard time blocking them...

         

designaweb

3:48 pm on Nov 18, 2004 (gmt 0)

10+ Year Member



Hi,

I have a containing adresses of companies, and yes, I have collected these myself - didn't scrape them. The thing is, we are being scraped by lot's of IP's. I can block them easily by adding

Deny from 123.123.123.123

in my .htaccess file, but I am sure that some of the punks get my data before I realise they have been scraping my data.

Is there anything I can do to prevent scraping? Perhaps limit the amount of pages an IP can view per second, per minute, per hour, per day, per week, etc.?

I'd love some feedback on this one...

Matt Probert

3:49 pm on Nov 18, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you have your own server, speak to your host about iptables....

Matt

uncle_bob

4:36 pm on Nov 18, 2004 (gmt 0)

10+ Year Member



A common way to catch scrapers is to implement some form of bot-trap script that adds the ip to htaccess whenever it gets tripped.

A quick search should throw up plenty of examples such as
[webmasterworld.com...]