| robots.txt
Where does it go? |
zoobie
msg:1527924 | 3:54 am on Sep 19, 2003 (gmt 0) |
I have several subdomains and need to use a robots.txt file. Where would I put it? In the tld or each subdomain's folder? What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?
Thanks
|
dougmcc1
msg:1527925 | 2:25 pm on Sep 19, 2003 (gmt 0) |
The root directories of the subdomain and tld. | What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php? |
|
In your subdomain:
User-agent: *
Disallow: /mailer/mail.php In your tld root directory:
User-agent: *
Disallow: /subdomain/mailer/mail.php Remember to put a blank line at the end of your robots.txt and use a validator to make sure you dont have any errors.
|
DerekJPreston
msg:1527926 | 3:17 pm on Sep 19, 2003 (gmt 0) |
Google support the presence of robots.txt files in other directories, I believe, but only as a way of (temporarily) removing files. You have to tell them where the buried robots.txt files are located. By default, they like other robots only look in the root of each site. (www.yoursite.com/robots.txt) Regards, Derek J. Preston [edited by: agerhart at 6:02 pm (utc) on Sep. 19, 2003]
[edit reason] no sigs please [/edit]
|
zoobie
msg:1527927 | 5:58 pm on Sep 19, 2003 (gmt 0) |
In both the subdomain and tld's root? Ok.
A blank space at the bottom? My text editor removes the white space and it validated fine...
Comments?
|
|
|
