Welcome to WebmasterWorld Guest from 54.161.201.189

Forum Moderators: goodroi

robots.txt

Where does it go?

   
3:54 am on Sep 19, 2003 (gmt 0)

10+ Year Member



I have several subdomains and need to use a robots.txt file. Where would I put it? In the tld or each subdomain's folder? What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?
Thanks
2:25 pm on Sep 19, 2003 (gmt 0)

10+ Year Member



Where would I put it?

The root directories of the subdomain and tld.

What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?

In your subdomain:
User-agent: *
Disallow: /mailer/mail.php

In your tld root directory:
User-agent: *
Disallow: /subdomain/mailer/mail.php

Remember to put a blank line at the end of your robots.txt and use a validator to make sure you dont have any errors.

3:17 pm on Sep 19, 2003 (gmt 0)

10+ Year Member



Google support the presence of robots.txt files in other directories, I believe, but only as a way of (temporarily) removing files. You have to tell them where the buried robots.txt files are located. By default, they like other robots only look in the root of each site. (www.yoursite.com/robots.txt)

Regards,

Derek J. Preston

[edited by: agerhart at 6:02 pm (utc) on Sep. 19, 2003]
[edit reason] no sigs please [/edit]

5:58 pm on Sep 19, 2003 (gmt 0)

10+ Year Member



In both the subdomain and tld's root? Ok.
A blank space at the bottom? My text editor removes the white space and it validated fine...
Comments?
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month