homepage Welcome to WebmasterWorld Guest from 54.166.228.100
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Subdomains
How to exclude
gosman




msg:1526131
 6:35 pm on Jan 3, 2004 (gmt 0)

I have a subdomain pointing at an affilite server.

Example:
reservations.mysite.com > points to affiliate servers IP address. This is set by my ISP so I have no control over this subdomain.

Is it possible to exclude this subdomain from being spidered by using the www.mysite.com/robots.txt If so what would the Disallow statement look like.

Thank you in advance

 

closed




msg:1526132
 1:09 am on Jan 6, 2004 (gmt 0)

You should just put a robots.txt document in the directory referred to by reservations.mysite.com and treat it like it's its own domain.

gosman




msg:1526133
 8:39 am on Jan 6, 2004 (gmt 0)

Hi Closed.

I don't have control over these subdomains. They have been set up by my ISP.

If I put the following statements

Disallow: /reservations.mysite.com/
Disallow: /guides.mysite.com/

In the robots.txt on mysite.com. Will this prevent links to these subdomains being spidered?

closed




msg:1526134
 2:54 pm on Jan 6, 2004 (gmt 0)

To answer your question, no.

Before crawling reservations.mysite.com, the robot will ask the server for reservations.mysite.com/robots.txt, not www.mysite.com/robots.txt. So what you should do is make a new robots.txt specifically for that subdomain, and put it in the same directory as reservations.mysite.com/index.html (or index.php or whatever your default page is).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved