robots.txt to stop some domains

How to allow only one domain to be indexed for the same website.

[john_pinx]

I have one website with 5 domain names pointing at it. This is bad for search engines. Can I use robots.txt to restrict the robots looking at a domain which I don't want? e.g. mydomain.com is the one I want and I submit it for indexing, mydomain.org and mydomain.net are pointing at the same website, but I don't want them to be indexed.

Tks in advance :-)

JohnP

[encyclo]

This is not something you can do with robots.txt alone.

You have two choices. The first is to cloak your robots.txt and allow or disallow dependent on the http_host header. It's not impossible to do, but it's hard work.

The second and much easier choice is to use mod_rewrite to send a 301 Permanent Redirect for any request for a domain which is not the main one. In that way, your content only appears under one domain, and so can only be indexed once.

[john_pinx]

Thanks -

> The second and much easier choice is to use mod_rewrite to send a 301 Permanent Redirect for any request for a domain which is not the main one. In that way, your content only appears under one domain, and so can only be indexed once.

OK - so I get mod_rewrite (somehow - never done that before) and it sends a redirect if a .net or .org domain is used. I presume this allows human browsers to see the site and robots to be told that it's not available?

Slainte.

[epicadventures]

I've got the same question, someone please enlighten us on how to do this. I'd like to let yahoo slurp index one of the domains, but not the other, and google to index the one that yahoo doesn't index. Is this possible. Google has already indexed a few of the pages from my second domain that I just set up and they rank higher than the same pages from my main domain, even though I only have one incoming link for that domain. Is this bad or good. I didn't really want google to index that domain because of concerns with duplicate content.

[jdMorgan]

A search [google.com] will turn up much previous discussion of this and related subjects.

Jim