Page is a not externally linkable
Ujang - 9:24 am on Jul 7, 2010 (gmt 0)
Ok, blocking "bot search" may be a little drastic... Does anyone know if any genuine bot has that phrase in its UA?
How about to just allow several popular bot to access your site? Additionally one could perform DNS reverse lookup to ensure the origin of that bot.
popular UA: [user-agents.org...]
how to verify bot: [google.com...]
----
Below could be another way to identify a bad bot..
source: Google Webmaster Blog..In order to fetch from the "official" Googlebot IP range, the bot has to respect robots.txt and our internal hostload conventions so that Google doesn't crawl you too hard.
then you might create hidden link > block it via robot.txt > see if any bot reach that link > get the ip > block it!
just my 2cents