Page is a not externally linkable
tangor - 11:24 am on Jan 11, 2010 (gmt 0)
On the other hand if you are allowing an obscure, unknown SE to spider and do not want to reveal that info... I don't have any simple suggestions. Robots.txt, by definition, should be available to any and all requests.
Not sure why you'd want to restrict access to robots.txt. If you are running a white list robots.txt, ie, allow the ones you want and disallow all the rest, the only thing a competitor could see is what they already have in their OWN robots.txt.... after all they want google, bing, and yahoo, too!