homepage Welcome to WebmasterWorld Guest from 54.237.54.83
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt
Where does it go?
zoobie




msg:1527924
 3:54 am on Sep 19, 2003 (gmt 0)

I have several subdomains and need to use a robots.txt file. Where would I put it? In the tld or each subdomain's folder? What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?
Thanks

 

dougmcc1




msg:1527925
 2:25 pm on Sep 19, 2003 (gmt 0)

Where would I put it?

The root directories of the subdomain and tld.

What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?

In your subdomain:
User-agent: *
Disallow: /mailer/mail.php

In your tld root directory:
User-agent: *
Disallow: /subdomain/mailer/mail.php

Remember to put a blank line at the end of your robots.txt and use a validator to make sure you dont have any errors.

DerekJPreston




msg:1527926
 3:17 pm on Sep 19, 2003 (gmt 0)

Google support the presence of robots.txt files in other directories, I believe, but only as a way of (temporarily) removing files. You have to tell them where the buried robots.txt files are located. By default, they like other robots only look in the root of each site. (www.yoursite.com/robots.txt)

Regards,

Derek J. Preston

[edited by: agerhart at 6:02 pm (utc) on Sep. 19, 2003]
[edit reason] no sigs please [/edit]

zoobie




msg:1527927
 5:58 pm on Sep 19, 2003 (gmt 0)

In both the subdomain and tld's root? Ok.
A blank space at the bottom? My text editor removes the white space and it validated fine...
Comments?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved