If I have a couple of subdomains - www.subdomain.mydomain.com - how do I keep the spiders from accessing fx. www.subdomain.mydomain.com/cgi-bin/ ? Do I use multiple robots.txt's - one for each subdomain, or can I control it all from the top level (www.mydomain.com/robots.txt)?
I have access to all areas of the domain/server.
TIA
