homepage Welcome to WebmasterWorld Guest from 54.211.157.103
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
How can I disallow google from a DN that mirrors a site
Can't use robots.txt file on just a domain name
wellnesscafe




msg:49806
 10:08 pm on Feb 25, 2003 (gmt 0)

I have a website "a.com". I am using a robots.txt file on that site to dissalow google from certain directories.

I also have a domain name "b.com" which mirrors "a.com".

'b.com' does not have a server. It is only a domain name that duplicates/mirrors 'a.com'

Per google's policy, I must not allow google to index a duplicate site. How do I dissalow google from indexing b.com?

Someone wrote "The site that you want to 'disallow' Google from indexing should be set up with a 301 Permanent Redirect to the other domain. You can also set this up via the DNS, but a 301 permanent redirect is preferable because the 'mirror' should not be listed in the search engines, only the main site."

I'm not sure I understand this and my server's support doesn't seem to know about it either.

Any advice?

 

WebGuerrilla




msg:49807
 10:21 pm on Feb 25, 2003 (gmt 0)


If you are on an Apache server with mod rewrite installed, you can put this in your .htaccess file

RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_HOST} noindexdomain.com$
RewriteRule ^(.+) [indexdomain.com...] [L,R=301]

That will prevent a spider from indexing the first domain. If you can't use mod rewrite another solution is to use absolute urls for all your links. That won't prevent you homepage from being indexed twice, but it will prevent the rest of your site from being duplicated.

wellnesscafe




msg:49808
 3:52 am on Feb 26, 2003 (gmt 0)

Thank you WebGuerrilla. That is great info :)

Robert Charlton




msg:49809
 7:31 am on Feb 26, 2003 (gmt 0)

Because of linking issues that will inevitably arise from mirror sites, I'd advise against them. And other engines don't handle them as well as Google does.

It sounds like you may simply be pointing multiple domains at the same site... getting mirrors as a side effect. This is an issue I was confronted with. I got some very good feedback about handling the situation on this thread:

Pointing multiple domain names to main site without mirrors
How to do this without hosting them separately and using 301s?
[webmasterworld.com...]

Mel




msg:49810
 11:09 am on Feb 26, 2003 (gmt 0)

How does your b site mirror your a site if it has no server? If I am reading you right your b site is "parked" to your a site, and thus should not be a problem at all.

wellnesscafe




msg:49811
 2:47 pm on Feb 26, 2003 (gmt 0)

I see your point Robert. I'm glad I checked this out before pointing 'b.com' to 'a.com'. Right now 'b.com' is only a forwarding DN and I will keep it that way to avoid getting mirror effect and inbound linking problems.

Mel, I'm a dilettante at all this, confused as can be, and trying to learn stuff. Basically the 'b.com' is not a site. It is just a forwarded domain name. I had been thinking about having it mirror 'a.com'. But I've changed my mind.

wellnesscafe




msg:49812
 6:57 pm on Feb 26, 2003 (gmt 0)

One more question: google and domain name forwarding

I have several domain names that are currently being forwarded to my website. Does google consider forwarding DN the same as mirroring?

Upon their usage, these domain names automatically change/reverts back to the original DN. For example, let's say my original domain name is my-domain-name.com. When someone types in a forwarded domain name such as 'mydomain-name.com or 'mydomainname.com' it automatically change/convert back to my-domain-name.com .

I don't see how google can index forwarding DN or consider these as mirrors. But I have to double check with you guys. I just don't want to do anything that could possibly violate google's policy in the slightest way.

argots




msg:49813
 8:16 pm on Feb 26, 2003 (gmt 0)

A question about mirroring:
My main site is like: www.mydomain.com.
I test my changes at test.mydomain.com.

Would Google consider this to be a bad thing?

(test.mydomain.com is almost always an exact copy of the main site, except when changes are being tested)

wellnesscafe




msg:49814
 8:55 pm on Feb 26, 2003 (gmt 0)

I don't understand your message argots. Can you please explain about this testing thing. How do you test a domain name? What do you mean by "when changes are being tested"?

werty




msg:49815
 9:28 pm on Feb 26, 2003 (gmt 0)

My question is that are all the paths the same?

Like if you go to domain1.com/file.html can you type in domain2.com/file.html and get to the same page?

I have used domain fwding in the past and those would do domain2.com as a frame. and it would load domain1.com

so what would happen is that www.domain2.com could be entered into the browser. it would open up a frame around www.domain1.com. the url in the browser would remain at www.domain2.html but all the content would be coming from www.domain1.html. so you could actually be at www.domain1.com/photos/gallery.html and the browser would still display www.domain2.com as the url.

hopefully i made some sense.

wellnesscafe




msg:49816
 10:46 pm on Feb 26, 2003 (gmt 0)

Well, Werty, I'm not an expert on this but I think that this forwarding setting depends on which company registers your domain name. That is why I am transferring my domain name now to a different company. The new company offers much better forwarding than my previous company. At godaddy I can choose the setting called "masking" if I wish. When you maks a domain name, you don't see the original URL. For example, if the original url is yourdomain-xyz.com and you want to mask it with the forwarding domain yourdomain-abc.com that your visitors will only see yourdomain-abc.com. The new company offers much more flexibility. I'm very happy with them.

werty




msg:49817
 11:28 pm on Feb 26, 2003 (gmt 0)

So you now switched to godaddy? If so that is who I use for a few of my domains.

Some of my older domains were on domaindirect and they are the ones who used the funky frames pages.

I think what you are planning to do should be safe. Let me know the results because I had a few ideas I wanted to do along the same lines.

argots




msg:49818
 7:32 pm on Feb 27, 2003 (gmt 0)

I don't understand your message argots. Can you please explain about this testing thing. How do you test a domain name? What do you mean by "when changes are being tested"?

I have two exact copies of the site. One is the "real" site and one is where I try-out new ideas for the site. If my domain is: mydomain.com, then one copy is at www.mydomain.com and the other one is at test.mydomain.com.

My question is: If Google manages to index test.mydomain.com, and they already know about www.mydomain.com, then will they have a problem with this?

jdMorgan




msg:49819
 8:19 pm on Feb 27, 2003 (gmt 0)

argot,

Generally, Google will pick one domain name - usually the one with the highest PR - and use that one.

However, to avoid nasty surprises, I suggest your disallow all robots from your test subdomain using robots.txt.

What you don't want to do is try to promote multiple domains all pointing to the same content. If they find lots of links to both domains, some SEs may interpret that as domain-name spamming, and you also end up "splitting" your page rank and link pop across multiple domains.

Jim

wellnesscafe




msg:49820
 9:39 pm on Feb 27, 2003 (gmt 0)

jdMorgan: I'm so glad you brought this up because I was concerned about having 2 domain names. When you say "If they find lots of links to both domains" do you mean that if mydomain-1.com has lots of links to mydomain-2.com and vise versa?

What if I have 2 domain names with identical sites (each with their own individual hosting) and if had no links between each domain/sites, will google consider this domain spamming anyway?

What do you mean by "and link pop across multiple domain"?

werty: I've had several forwarding domain names with godaddy for a couple of years and have just transfered another one. I think godaddy offers superior services. For example, the forwarding with the former registrar only took the forwarding domain name to the home page. If you typed mydomainname.com/filename.htm it brings up an error message. That was a big problem for me. With godaddy, the forwarding takes the visitor to the specific file name page. Isn't that neat or what!

What results do you want me to share with you? I don't get it.

jdMorgan




msg:49821
 10:35 pm on Feb 27, 2003 (gmt 0)

wellnesscafe,

If you actively seek and get links to two domains with the same content, and an SE rep looks at your site(s) as the result of a complaint, I'd guess you would be PR0'ed.

I want to say right up front that the only reason I ever have two or more domain names pointed to the same site is in order to protect 'branding' of the site. IOW, I've registered the .com, .org, and sometimes the .net of the domain name, and sometimes the hyphenated and non-hyphenated variants as well. At the same time, my approach to SEO is "long-term" - In many cases, these are sites with which I have only a short-term contract with no maintenace clause, so I forego any techniques which might cause them trouble long-term. However, my basic attitude is "each to his own" and "you do what you think best," so don't take me for a fanatical "Mr. Clean-SEO" evangelist. I think it's a good idea for long-term success, but I'm not preaching, OK?

That said, when using multiple domain names, I promote only one of them, and use a 301-Permanent redirect from the alternate domains to the "official" domain.

This prevents any future complaints about domain spamming (multiple listings in the SERPs for duplicate or essentially identical content) and prevents diluting Page Rank and Link Popularity and all their derivatives (e.g. hub and authority ranks) by concentrating all incoming links onto only one domain.

With the point of view I described above in mind, I would never cross-link two sites with more than a few (OK, a dozen links on a 100-page site, for example) links if these two sites were related by ownership at any level, and if they offered either substantially identical or totally unrelated content. I do indeed cross-link multiple sites which I work on, but only if it is a benefit for the user. I've just read too many posts here about cross-linking penalties here to ever try that anyway (OK, call me chicken).

If I understand Google's viewpoint (and I think I do), they would see two listings in the SERPs for two different domains serving up the same content as blatant spamming. The duplicate listing offers no advantage to the searcher, and pushes some other worthy site down off the page (a point where your needs and the SE's needs may admittedly be in complete opposition).

At the same time, it is pretty much inevitable if you have multiple domains that a few webmasters will link to your site using the 'wrong' domain name, and I wouldn't worry too much about a few of these links.

Another point in the "full disclosure" vein... I am strictly small-time and have not done hundreds of sites, or any sites with more than a few hundred pages. I have never tried to promote a site about Jennifer Lopez to the number one spot, either. All I feel I can contribute here is a fairly good understanding of the underlying technology of search, URL redirection, and client/server interactions, along with an understanding of "Google's ethics of the Web." So, in short form, "Your mileage may vary - A Lot!"
However, I never lose any sleep during the updates, either. :)

Again, not evangelizing here - these are just my opinions.

Now I'd better go see if anyone needs help with mod_rewrite in the forums where I belong. :)

Jim

Bradley




msg:49822
 11:20 pm on Feb 27, 2003 (gmt 0)


This discussion is very interesting to me, cause I have a question along these lines. My business is partnering with another company. The other company is creating a private label site for us. 99% of the HTML and functionality is the same. The main difference is their log has been replaced by our logo on the private label site.....

How will the major search engines react to this private label site? The other company has their own website, and they are also creating a private label site (on a separate IP address) using our graphics. This private label site is a major part of our strategy and we are seeking to build a brand name behind it (that's why we decided to create this site versus using an affiliate link to their company website).

Should we promote this private label site to the major search engines and directories? We'd like to market this site on the internet because it is a valuable part of our business model. On the other hand, we don't want the site to be penalized because its seen as duplicative.

Any suggestions and/or ideas?

Robert Charlton




msg:49823
 6:32 am on Feb 28, 2003 (gmt 0)

Bradley - I have a non-profit client that had co-branded mirror sites... using subdomains... with some major newspapers as a way of attracting "walk-in traffic." Because some of these were in fact major newspapers, the mirror sites started outranking the main site for some terms.

Though it may seem foolish in straightforward SEO terms (search engine traffic at all costs), I ultimately suggested that the robots.txt block all spiders from the subdomains.

I could see that this situation would otherwise very quickly have caused possible duplicate content penalties, a loss of ranking in the main site, and very likely a loss of brand ID on the web.

As for multiple domains pointing to the same site, again, I'm strongly against duplicate content, very definitely against mirrored content, wherever possible. While Google can usually sort this out satisfactorially, many of the other engines can't... and, wherever I've seen it stay unchecked, it ultimately leads to grief.

wellnesscafe




msg:49824
 10:56 pm on Feb 28, 2003 (gmt 0)

Jim - Thank you so much for taking the time to explain this information in detail. It is very helpful. I have found some interesting information and guidance about 301-permament redirect at [webmasterworld.com...] and [webmasterworld.com...] I certainly don't call you chicken...rather cautious and wise. Like you, I don't want to loose sleep over google. Much rather be safe than sorry. Your point is well made and I value your opinion.

jdMorgan




msg:49825
 12:54 am on Mar 1, 2003 (gmt 0)

wellnesscafe,

No problem... Hope it helps.

For a good introduction to mod_rewrite, which allows "industrial strength" redirects, see this Introduction to mod_rewrite [webmasterworld.com] post.

Note that the link to [webmasterworld.com...] above needs to have the period on the end removed to work.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved