shri - 7:02 am on Nov 24, 2005 (gmt 0)

Brett,

Have you considered "cloaking" your robots.txt?

You can force robots.txt to be a php file and run all sorts of interesting queries on a mysql DB before you spit the output out.

e.g

Accepts cookies: send it a disallow all.
Comes from bad ip range: send it a disallow all.
Comes from a legit G/Y/M DC: send it a proper robots.txt

Another option might be to log to mysql and have a cron job that queries the IP addresses every x minutes and reconfigs your iptable / firewall / deny lists.

(yeah yeah I know you also need to run a business... but I'm sure the geek inside you is yelling to find a more elegant solution!)