Forum Moderators: goodroi
Example:
reservations.mysite.com > points to affiliate servers IP address. This is set by my ISP so I have no control over this subdomain.
Is it possible to exclude this subdomain from being spidered by using the www.mysite.com/robots.txt If so what would the Disallow statement look like.
Thank you in advance
Before crawling reservations.mysite.com, the robot will ask the server for reservations.mysite.com/robots.txt, not www.mysite.com/robots.txt. So what you should do is make a new robots.txt specifically for that subdomain, and put it in the same directory as reservations.mysite.com/index.html (or index.php or whatever your default page is).