Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

Where does it go?

     
3:54 am on Sep 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 20, 2002
posts:179
votes: 0


I have several subdomains and need to use a robots.txt file. Where would I put it? In the tld or each subdomain's folder? What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?
Thanks
2:25 pm on Sept 19, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 28, 2003
posts:480
votes: 1


Where would I put it?

The root directories of the subdomain and tld.

What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?

In your subdomain:
User-agent: *
Disallow: /mailer/mail.php

In your tld root directory:
User-agent: *
Disallow: /subdomain/mailer/mail.php

Remember to put a blank line at the end of your robots.txt and use a validator to make sure you dont have any errors.

3:17 pm on Sept 19, 2003 (gmt 0)

New User

10+ Year Member

joined:Aug 12, 2002
posts:18
votes: 0


Google support the presence of robots.txt files in other directories, I believe, but only as a way of (temporarily) removing files. You have to tell them where the buried robots.txt files are located. By default, they like other robots only look in the root of each site. (www.yoursite.com/robots.txt)

Regards,

Derek J. Preston

[edited by: agerhart at 6:02 pm (utc) on Sep. 19, 2003]
[edit reason] no sigs please [/edit]

5:58 pm on Sept 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 20, 2002
posts:179
votes: 0


In both the subdomain and tld's root? Ok.
A blank space at the bottom? My text editor removes the white space and it validated fine...
Comments?
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members