Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Subdomains How to exclude gosman
I have a subdomain pointing at an affilite server.
reservations.mysite.com > points to affiliate servers IP address. This is set by my ISP so I have no control over this subdomain.
Is it possible to exclude this subdomain from being spidered by using the www.mysite.com/robots.txt If so what would the Disallow statement look like.
Thank you in advance
You should just put a robots.txt document in the directory referred to by reservations.mysite.com and treat it like it's its own domain. gosman
I don't have control over these subdomains. They have been set up by my ISP.
If I put the following statements
In the robots.txt on mysite.com. Will this prevent links to these subdomains being spidered?
To answer your question, no.
Before crawling reservations.mysite.com, the robot will ask the server for reservations.mysite.com/robots.txt, not www.mysite.com/robots.txt. So what you should do is make a new robots.txt specifically for that subdomain, and put it in the same directory as reservations.mysite.com/index.html (or index.php or whatever your default page is).