Welcome to WebmasterWorld Guest from 184.108.40.206 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor
Subdomains How to exclude gosman msg:1526131 6:35 pm on Jan 3, 2004 (gmt 0) I have a subdomain pointing at an affilite server.
reservations.mysite.com > points to affiliate servers IP address. This is set by my ISP so I have no control over this subdomain.
Is it possible to exclude this subdomain from being spidered by using the www.mysite.com/robots.txt If so what would the Disallow statement look like.
Thank you in advance
closed msg:1526132 1:09 am on Jan 6, 2004 (gmt 0)
You should just put a robots.txt document in the directory referred to by reservations.mysite.com and treat it like it's its own domain. gosman msg:1526133 8:39 am on Jan 6, 2004 (gmt 0)
I don't have control over these subdomains. They have been set up by my ISP.
If I put the following statements
In the robots.txt on mysite.com. Will this prevent links to these subdomains being spidered?
closed msg:1526134 2:54 pm on Jan 6, 2004 (gmt 0)
To answer your question, no.
Before crawling reservations.mysite.com, the robot will ask the server for reservations.mysite.com/robots.txt, not www.mysite.com/robots.txt. So what you should do is make a new robots.txt specifically for that subdomain, and put it in the same directory as reservations.mysite.com/index.html (or index.php or whatever your default page is).