Welcome to WebmasterWorld Guest from 54.157.222.62

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and subdomains.

How do spiders react on robots.txt on subdomains?

   
11:05 am on Jan 29, 2001 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



If I have a couple of subdomains - www.subdomain.mydomain.com - how do I keep the spiders from accessing fx. www.subdomain.mydomain.com/cgi-bin/ ?

Do I use multiple robots.txt's - one for each subdomain, or can I control it all from the top level (www.mydomain.com/robots.txt)?

I have access to all areas of the domain/server.

TIA