Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt


dougmcc1 - 2:25 pm on Sep 19, 2003 (gmt 0)


Where would I put it?

The root directories of the subdomain and tld.

What would the syntax look like for disallowing, say, subdomain.domain.com/mailer/mail.php?

In your subdomain:
User-agent: *
Disallow: /mailer/mail.php

In your tld root directory:
User-agent: *
Disallow: /subdomain/mailer/mail.php

Remember to put a blank line at the end of your robots.txt and use a validator to make sure you dont have any errors.


Thread source:: http://www.webmasterworld.com/robots_txt/1.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com