Welcome to WebmasterWorld Guest from 18.104.22.168
I would like to auto add such ips to my .htaccess file "deny from xx.xx.xx.xx" statement.
But I would still like to keep specific bots like googlebots, etc from crawling.
How do I do this?
Install AlexK's modified version of xlcus' php script to ban runaway crawlers [webmasterworld.com]. This script detects excessively-fast consecutive page requests.
Adding exclusions to avoid banning *any* major 'bot is a good idea.