lucy24 - 2:14 am on Feb 11, 2013 (gmt 0)
So that's why bing/msn's rate of robots.txt requests has dropped so dramatically on my site since last year. They've decided they like your version better.
If you had your own server you could implement some nasty business with firewalls. But on htaccess your only real options are to let them in or not let them in.
:: quick detour to any old log entry ::
ymmv, but in my setup, handing over robots.txt uses fewer bytes than a 403. I do not understand why the numbers are not exactly the same each time --that is, a consistent number for robots.txt and a consistent number for 403-- and I'm fairly certain I would not understand the explanation.