robots.txt

Where does it go?

[zoobie]

I have several subdomains and need to use a robots.txt file. Where would I put it? In the tld or each subdomain's folder? What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?
Thanks

[dougmcc1]

Where would I put it?

The root directories of the subdomain and tld.

What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?

In your subdomain:
User-agent: *
Disallow: /mailer/mail.php

In your tld root directory:
User-agent: *
Disallow: /subdomain/mailer/mail.php

Remember to put a blank line at the end of your robots.txt and use a validator to make sure you dont have any errors.

[DerekJPreston]

Google support the presence of robots.txt files in other directories, I believe, but only as a way of (temporarily) removing files. You have to tell them where the buried robots.txt files are located. By default, they like other robots only look in the root of each site. (www.yoursite.com/robots.txt)

Regards,

Derek J. Preston

[edited by: agerhart at 6:02 pm (utc) on Sep. 19, 2003]
[edit reason] no sigs please [/edit]

[zoobie]

In both the subdomain and tld's root? Ok.
A blank space at the bottom? My text editor removes the white space and it validated fine...
Comments?