| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|Robots.txt and subdomains.|
How do spiders react on robots.txt on subdomains?
| 11:05 am on Jan 29, 2001 (gmt 0)|
If I have a couple of subdomains - www.subdomain.mydomain.com - how do I keep the spiders from accessing fx. www.subdomain.mydomain.com/cgi-bin/ ?
Do I use multiple robots.txt's - one for each subdomain, or can I control it all from the top level (www.mydomain.com/robots.txt)?
I have access to all areas of the domain/server.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved