Msg#: 3289506 posted 12:33 pm on Mar 22, 2007 (gmt 0)
I have a huge website which is dynamically generated. I have noticed over a period of time that there are stupid web downloaders, etc which have been aggressively downloading my website thus enormously increasing the load on the server and slowing down my site.
I would like to auto add such ips to my .htaccess file "deny from xx.xx.xx.xx" statement.
But I would still like to keep specific bots like googlebots, etc from crawling.