Pfui - 11:35 pm on Jul 17, 2011 (gmt 0)
For me, site-specific access control on my old Apache box is via .htaccess, and relying primarily on mod_rewrite (plus some Deny from).
robots.txt is a toothless tiger if it's neither retrieved nor heeded aprés retrieval. I use it, but only via a script that gives the 'full' list to my choice of major SEs, and a full Disallow to absolutely anyone else.
The script I use is a verrry heavily modified version of Lee Killough's officially outdated robots.txt-generating CGI [leekillough.com...]