| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Robots.txt and subdomains.|
How do spiders react on robots.txt on subdomains?
If I have a couple of subdomains - www.subdomain.mydomain.com - how do I keep the spiders from accessing fx. www.subdomain.mydomain.com/cgi-bin/ ?
Do I use multiple robots.txt's - one for each subdomain, or can I control it all from the top level (www.mydomain.com/robots.txt)?
I have access to all areas of the domain/server.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved