How to deal with duplicate sites

duplicate sites, google seo

[johnlim9988]

Hi,

We have several duplicate sites. We created the duplicate sites (similar sites) about two years ago. It has some traffic for the first yeat. Then don't have traffic for the following year until now.

We concentrated on one site about one year. We write a lot of unique contes for the site but still not traffic.

Did the google ban us already because of duplicate sites. what should I do? Reinclusin report? We don't want to shut down the sites as it still have a little traffic.

Pls kindly advise.

[shogun_ro]

My advice:

1.Delete all interlinks between them.
2.Concentrate on all sites and change content till 60% duplicate is reached.

[johnlim9988]

1. all the interlinks are deleted long time ago. (about one year ago).

2. Very difficult to write so many contents for all the sites, just can only concentrate one site. How to do with other sites?

[hutcheson]

Don't try to do more than you can do well. There's enough badly-done cruft on the web already.

So it's a no-brainer decision to kill the other sites. Permanent-redirect them to the one site you can actually support.

[SeoCatfish]

use a 301 redirect on all sites that you want to consolidate and point them to your main site. This will transfer your link popularity and Page Rank as well.

[johnlim9988]

It is very difficult to do redirect, as the file name structures are not the same.

How about I just concentrate on one website and forbidden googlebot to crawl all the other sites. Let all the other sites die by themselves gradually.

Will this save my main site with many unique contents? Now the problem is even we write over 200 unique articles at the main site and still cannot rank at google as before.

[johnlim9988]

Anyboday have any real experience? And hwo to recover the main site? Is it ok to just forbidden the robots to crawl all the non-important sits? (which has duplicate contents.)

[tedster]

Google prefers that you use robots.txt and the robots meta tag to direct them away from indexing duplicate content and toward your preferred urls. You can even speed up the process by using the url removal tool that Google provides at [services.google.com...] - I've done this succesfully many times for clients.

Whether that will be enough to recover your main domain in the SERPs is still to be seen, but it is a good first step.

[edited by: tedster at 3:21 am (utc) on Feb. 12, 2007]

[myvoice]

I ve changed the content of a category page which previously had the content of the sub category page three weeks back for which I ve seen Yahoo and Live to crawl the new content while Google still shows the old subcategory page content. Is Google page removal tool helpful in this case?

[johnlim9988]

I am thinking among the following two ways, which is the best way

1) Use robots.txt to the duplicate sites to ban all robots. Google have removal tool, did yahoo and msn have? anybody know?

2) Use 301 redirect. have any way to redirec the whole site to another site? for example, www.domain1.com and www.domain1.com/sub1/page1.html etc all redirect to maindonain.com?

which way is the best way? Let's compare the two methods,

method 1) if somebody bookmark domain1.com then he can still directly go to domain1.com/page2.html, if use method 2 then the guest have no way to bookmarked page.

method 2) if use 301 redirect then the links to domain1.com can point to maindomain.com then it is good for maindomain.com, right?

Anyone have any experience which way can let the mainsite to go out of the duplicate penaluty?

Now we have no hope for the dupicate sites, we will be happy just the main site go out of the penalty.

Thanks.

[johnlim9988]

Did somebody has any real experience and good idea?
Thanks.

[activeco]

Tedster already gave you a good idea.
Have you tried to implement it?

Anyway, what exactly is the problem?
Are you saying that BOTH the sites are completely out of the index, one site, or just some pages?
In case of duplicates Google usually choose one url as the original and keep it in the main index. If all (original and duplicates) belongs to you and at least one copy is indexed then you have no problem.

[johnlim9988]

1. already take tedster's advise to remove the duplicated sites (except the main site) from google at [services.google.com...]

2. Use robots.txt to ban all the robots to all the duplicated sites (excpet the main site.)

3. Only concentrate on the main website.

Do you think the above is the best way? or have any other better way?

[tedster]

If you've already used the removal tool, those urls are now gone from Google for 180 days - no backing out is possible. But you've also greatly simplified your situation now, so all your further troubleshooting will be a lot easier.

If you haven't already, I'd suggest studying the threads in our Google Hot Topics [webmasterworld.com] at a minimum. I've pinned it to the top of the Google Search index page so it's easy to find.

[johnlim9988]

Hi,

is the following correct robots.txt to ban googlebot and yahoo sluep, can put two robots at one line? As this duplciated site still have some MSN traffic so we decide to ban google and yahoo.

*****************************************
User-agent: googlebot, Slurp
Disallow: /
*****************************************

Thanks,

[tedster]

Those specs for robots.txt can be inscrutable - someone involved should take a course in technical writing. But according to the lines I quoted below, I believe that what you have is correct for your purpose. IF the bots are written to follow the spec, of course.

User-agent
The value of this field is the name of the robot the record is describing access policy for.

If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.

[robotstxt.org...]

For my taste, I would feel safer doing separate rules for each bot. No offense meant to either Google or Yahoo engineers. It's just that the robots.txt standard reads so wonky in some areas that I like to play it as safe as possible - a couple extra lines of text will not cause big bandwidth problems, you know?

[edited by: tedster at 9:55 pm (utc) on Feb. 6, 2007]

[johnlim9988]

Is this better?

User-agent: googlebot
Disallow: /
User-agent: Slurp
Disallow: /

or this better?

User-agent: googlebot
User-agent: Slurp
Disallow: /

[tedster]

#1

[johnlim9988]

Did google have such function that if I have several duplicate sites and I only google to rank my main site and ignore other sites?

[g1smd]

>> #1 <<

Almost!

If you use #1 you MUST have a blank line before the next User-Agent line, for each extra User-Agent.

Other helpful comments in: [webmasterworld.com...]

[MThiessen]

It is very difficult to do redirect, as the file name structures are not the same.

Just use mod re-write and do the entire domains to the root of the site you are keeping, no need to worry over the individual pages, in fact you could even delete them all.

Don't know the exact code, I'm sure you can find it by searching here for "mod-rewrite redirect site".

[johnlim9988]

Hi,

After ban googlebot at robots.txt and use the removal tool, today I checked and no page for the duplicate sites indexed in google now.

My question, when will my unique site recover?

Thanks.