Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

How can I disallow google from a DN that mirrors a site

Can't use robots.txt file on just a domain name

10:08 pm on Feb 25, 2003 (gmt 0)

10+ Year Member

I have a website "a.com". I am using a robots.txt file on that site to dissalow google from certain directories.

I also have a domain name "b.com" which mirrors "a.com".

'b.com' does not have a server. It is only a domain name that duplicates/mirrors 'a.com'

Per google's policy, I must not allow google to index a duplicate site. How do I dissalow google from indexing b.com?

Someone wrote "The site that you want to 'disallow' Google from indexing should be set up with a 301 Permanent Redirect to the other domain. You can also set this up via the DNS, but a 301 permanent redirect is preferable because the 'mirror' should not be listed in the search engines, only the main site."

I'm not sure I understand this and my server's support doesn't seem to know about it either.

Any advice?

10:21 pm on Feb 25, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

If you are on an Apache server with mod rewrite installed, you can put this in your .htaccess file

RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_HOST} noindexdomain.com$
RewriteRule ^(.+) [indexdomain.com...] [L,R=301]

That will prevent a spider from indexing the first domain. If you can't use mod rewrite another solution is to use absolute urls for all your links. That won't prevent you homepage from being indexed twice, but it will prevent the rest of your site from being duplicated.

3:52 am on Feb 26, 2003 (gmt 0)

10+ Year Member

Thank you WebGuerrilla. That is great info :)
7:31 am on Feb 26, 2003 (gmt 0)

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Because of linking issues that will inevitably arise from mirror sites, I'd advise against them. And other engines don't handle them as well as Google does.

It sounds like you may simply be pointing multiple domains at the same site... getting mirrors as a side effect. This is an issue I was confronted with. I got some very good feedback about handling the situation on this thread:

Pointing multiple domain names to main site without mirrors
How to do this without hosting them separately and using 301s?


11:09 am on Feb 26, 2003 (gmt 0)

10+ Year Member

How does your b site mirror your a site if it has no server? If I am reading you right your b site is "parked" to your a site, and thus should not be a problem at all.
2:47 pm on Feb 26, 2003 (gmt 0)

10+ Year Member

I see your point Robert. I'm glad I checked this out before pointing 'b.com' to 'a.com'. Right now 'b.com' is only a forwarding DN and I will keep it that way to avoid getting mirror effect and inbound linking problems.

Mel, I'm a dilettante at all this, confused as can be, and trying to learn stuff. Basically the 'b.com' is not a site. It is just a forwarded domain name. I had been thinking about having it mirror 'a.com'. But I've changed my mind.

6:57 pm on Feb 26, 2003 (gmt 0)

10+ Year Member

One more question: google and domain name forwarding

I have several domain names that are currently being forwarded to my website. Does google consider forwarding DN the same as mirroring?

Upon their usage, these domain names automatically change/reverts back to the original DN. For example, let's say my original domain name is my-domain-name.com. When someone types in a forwarded domain name such as 'mydomain-name.com or 'mydomainname.com' it automatically change/convert back to my-domain-name.com .

I don't see how google can index forwarding DN or consider these as mirrors. But I have to double check with you guys. I just don't want to do anything that could possibly violate google's policy in the slightest way.

8:16 pm on Feb 26, 2003 (gmt 0)

10+ Year Member

A question about mirroring:
My main site is like: www.mydomain.com.
I test my changes at test.mydomain.com.

Would Google consider this to be a bad thing?

(test.mydomain.com is almost always an exact copy of the main site, except when changes are being tested)

8:55 pm on Feb 26, 2003 (gmt 0)

10+ Year Member

I don't understand your message argots. Can you please explain about this testing thing. How do you test a domain name? What do you mean by "when changes are being tested"?
9:28 pm on Feb 26, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

My question is that are all the paths the same?

Like if you go to domain1.com/file.html can you type in domain2.com/file.html and get to the same page?

I have used domain fwding in the past and those would do domain2.com as a frame. and it would load domain1.com

so what would happen is that www.domain2.com could be entered into the browser. it would open up a frame around www.domain1.com. the url in the browser would remain at www.domain2.html but all the content would be coming from www.domain1.html. so you could actually be at www.domain1.com/photos/gallery.html and the browser would still display www.domain2.com as the url.

hopefully i made some sense.

10:46 pm on Feb 26, 2003 (gmt 0)

10+ Year Member

Well, Werty, I'm not an expert on this but I think that this forwarding setting depends on which company registers your domain name. That is why I am transferring my domain name now to a different company. The new company offers much better forwarding than my previous company. At godaddy I can choose the setting called "masking" if I wish. When you maks a domain name, you don't see the original URL. For example, if the original url is yourdomain-xyz.com and you want to mask it with the forwarding domain yourdomain-abc.com that your visitors will only see yourdomain-abc.com. The new company offers much more flexibility. I'm very happy with them.
11:28 pm on Feb 26, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

So you now switched to godaddy? If so that is who I use for a few of my domains.

Some of my older domains were on domaindirect and they are the ones who used the funky frames pages.

I think what you are planning to do should be safe. Let me know the results because I had a few ideas I wanted to do along the same lines.

7:32 pm on Feb 27, 2003 (gmt 0)

10+ Year Member

I don't understand your message argots. Can you please explain about this testing thing. How do you test a domain name? What do you mean by "when changes are being tested"?

I have two exact copies of the site. One is the "real" site and one is where I try-out new ideas for the site. If my domain is: mydomain.com, then one copy is at www.mydomain.com and the other one is at test.mydomain.com.

My question is: If Google manages to index test.mydomain.com, and they already know about www.mydomain.com, then will they have a problem with this?

8:19 pm on Feb 27, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


Generally, Google will pick one domain name - usually the one with the highest PR - and use that one.

However, to avoid nasty surprises, I suggest your disallow all robots from your test subdomain using robots.txt.

What you don't want to do is try to promote multiple domains all pointing to the same content. If they find lots of links to both domains, some SEs may interpret that as domain-name spamming, and you also end up "splitting" your page rank and link pop across multiple domains.


9:39 pm on Feb 27, 2003 (gmt 0)

10+ Year Member

jdMorgan: I'm so glad you brought this up because I was concerned about having 2 domain names. When you say "If they find lots of links to both domains" do you mean that if mydomain-1.com has lots of links to mydomain-2.com and vise versa?

What if I have 2 domain names with identical sites (each with their own individual hosting) and if had no links between each domain/sites, will google consider this domain spamming anyway?

What do you mean by "and link pop across multiple domain"?

werty: I've had several forwarding domain names with godaddy for a couple of years and have just transfered another one. I think godaddy offers superior services. For example, the forwarding with the former registrar only took the forwarding domain name to the home page. If you typed mydomainname.com/filename.htm it brings up an error message. That was a big problem for me. With godaddy, the forwarding takes the visitor to the specific file name page. Isn't that neat or what!

What results do you want me to share with you? I don't get it.

10:35 pm on Feb 27, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


If you actively seek and get links to two domains with the same content, and an SE rep looks at your site(s) as the result of a complaint, I'd guess you would be PR0'ed.

I want to say right up front that the only reason I ever have two or more domain names pointed to the same site is in order to protect 'branding' of the site. IOW, I've registered the .com, .org, and sometimes the .net of the domain name, and sometimes the hyphenated and non-hyphenated variants as well. At the same time, my approach to SEO is "long-term" - In many cases, these are sites with which I have only a short-term contract with no maintenace clause, so I forego any techniques which might cause them trouble long-term. However, my basic attitude is "each to his own" and "you do what you think best," so don't take me for a fanatical "Mr. Clean-SEO" evangelist. I think it's a good idea for long-term success, but I'm not preaching, OK?

That said, when using multiple domain names, I promote only one of them, and use a 301-Permanent redirect from the alternate domains to the "official" domain.

This prevents any future complaints about domain spamming (multiple listings in the SERPs for duplicate or essentially identical content) and prevents diluting Page Rank and Link Popularity and all their derivatives (e.g. hub and authority ranks) by concentrating all incoming links onto only one domain.

With the point of view I described above in mind, I would never cross-link two sites with more than a few (OK, a dozen links on a 100-page site, for example) links if these two sites were related by ownership at any level, and if they offered either substantially identical or totally unrelated content. I do indeed cross-link multiple sites which I work on, but only if it is a benefit for the user. I've just read too many posts here about cross-linking penalties here to ever try that anyway (OK, call me chicken).

If I understand Google's viewpoint (and I think I do), they would see two listings in the SERPs for two different domains serving up the same content as blatant spamming. The duplicate listing offers no advantage to the searcher, and pushes some other worthy site down off the page (a point where your needs and the SE's needs may admittedly be in complete opposition).

At the same time, it is pretty much inevitable if you have multiple domains that a few webmasters will link to your site using the 'wrong' domain name, and I wouldn't worry too much about a few of these links.

Another point in the "full disclosure" vein... I am strictly small-time and have not done hundreds of sites, or any sites with more than a few hundred pages. I have never tried to promote a site about Jennifer Lopez to the number one spot, either. All I feel I can contribute here is a fairly good understanding of the underlying technology of search, URL redirection, and client/server interactions, along with an understanding of "Google's ethics of the Web." So, in short form, "Your mileage may vary - A Lot!"
However, I never lose any sleep during the updates, either. :)

Again, not evangelizing here - these are just my opinions.

Now I'd better go see if anyone needs help with mod_rewrite in the forums where I belong. :)


11:20 pm on Feb 27, 2003 (gmt 0)

10+ Year Member

This discussion is very interesting to me, cause I have a question along these lines. My business is partnering with another company. The other company is creating a private label site for us. 99% of the HTML and functionality is the same. The main difference is their log has been replaced by our logo on the private label site.....

How will the major search engines react to this private label site? The other company has their own website, and they are also creating a private label site (on a separate IP address) using our graphics. This private label site is a major part of our strategy and we are seeking to build a brand name behind it (that's why we decided to create this site versus using an affiliate link to their company website).

Should we promote this private label site to the major search engines and directories? We'd like to market this site on the internet because it is a valuable part of our business model. On the other hand, we don't want the site to be penalized because its seen as duplicative.

Any suggestions and/or ideas?

6:32 am on Feb 28, 2003 (gmt 0)

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Bradley - I have a non-profit client that had co-branded mirror sites... using subdomains... with some major newspapers as a way of attracting "walk-in traffic." Because some of these were in fact major newspapers, the mirror sites started outranking the main site for some terms.

Though it may seem foolish in straightforward SEO terms (search engine traffic at all costs), I ultimately suggested that the robots.txt block all spiders from the subdomains.

I could see that this situation would otherwise very quickly have caused possible duplicate content penalties, a loss of ranking in the main site, and very likely a loss of brand ID on the web.

As for multiple domains pointing to the same site, again, I'm strongly against duplicate content, very definitely against mirrored content, wherever possible. While Google can usually sort this out satisfactorially, many of the other engines can't... and, wherever I've seen it stay unchecked, it ultimately leads to grief.

10:56 pm on Feb 28, 2003 (gmt 0)

10+ Year Member

Jim - Thank you so much for taking the time to explain this information in detail. It is very helpful. I have found some interesting information and guidance about 301-permament redirect at [webmasterworld.com...] and [webmasterworld.com...] I certainly don't call you chicken...rather cautious and wise. Like you, I don't want to loose sleep over google. Much rather be safe than sorry. Your point is well made and I value your opinion.
12:54 am on Mar 1, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


No problem... Hope it helps.

For a good introduction to mod_rewrite, which allows "industrial strength" redirects, see this Introduction to mod_rewrite [webmasterworld.com] post.

Note that the link to [webmasterworld.com...] above needs to have the period on the end removed to work.