Welcome to WebmasterWorld Guest from 54.159.19.75

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and subdomains.

How do spiders react on robots.txt on subdomains?

     
11:05 am on Jan 29, 2001 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2530
votes: 1


If I have a couple of subdomains - www.subdomain.mydomain.com - how do I keep the spiders from accessing fx. www.subdomain.mydomain.com/cgi-bin/ ?

Do I use multiple robots.txt's - one for each subdomain, or can I control it all from the top level (www.mydomain.com/robots.txt)?

I have access to all areas of the domain/server.

TIA