DerekT - 5:24 am on Mar 8, 2003 (gmt 0)

If anyone would like to prevent "Web Copiers" or "Offline Browsers" without the need to update a .haccess file visit this thread for a great PHP solution.

[webmasterworld.com...]

It monitors page requests and if a user requests too many within a set timeframe, they are given a custom 503 message.

Initially I used a long .htaccess file to prevent these programs however, it didnt always work and I always had to add USER_AGENTS to the file when new programs were released. This also doesnt protect against these programs when people change their USER_AGENT to IE or Netscape.

Once I placed this script on my site, I caught 8 different people (unique) over a 24 hour period trying to leech my site. They all had normal browser USER_AGENT settings so a .htaccess wouldnt help. Since my site is all PHP and mySQL generated, this copying really hit my server hard. Some were requesting up to 17 pages a second!

Now that they are caught in realtime, my server is performing much better and my regular visitors are very happy.

If you visit the thread notice a few changes I added to ensure Googlebot is exempted from the limits and can request as many pages as it wishes.