Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
If it's possible to only allow robots to see this file, then what is the best method for doing this? :)
A simpler solution which I prefer to use myself is to use htaccess to block sensitive areas of my website. I use robots.txt more to deal with duplicate content issues.
You also can make a bot trap. Add a folder to the robots.txt file and do not list it anywhere else. Then wait and see what hits that folder and then ban that ip address. Since the only way to find that folder is from robots.txt you know it is a misbehaving bot or a hacker - either way you don't want it on your site.