How will Google treat redirecting these subdomains?

[sleidia]

Hi,

I had a dozen of subdomains that I used for geolocalization purpose, each one refering to a country and localized in GWT.

After a while I realized that the fact that all the subdomains content was 70% similiar would probably cause duplicate content (I wasn't aware of the duplicate content issue before .. and I still don't know if a rate of 70% can be considered as duplicate content by Google anyway).

So, I chose the safest route and unindexed all the subdomains with the Google URL removal tool. But I have those subdomains listed in web directories so I thought it would be a good idea to redirect them to my main domain like this in my htaccess :

RewriteCond %{HTTP_HOST} ^countryname.example.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/countryname/ [R=301,L]

Can someone confirm it's a good thing to do?

Thanks for the help :)

[tedster]

If all those subdomains were separately geo-targeted in Webmaster Tools, then you had no duplicate problem. I'd reverse the "de-indxing" and let it alone.

[sleidia]

Hi Tedster.

Thanks for the reply.

To be honest, I did quite an extensive research before de-indexing thoses subdomains. I've found several posts on Google groups and even an article written by a "search algorithm strategist" saying that duplicate content would arise from such similar subdomains. On every subdomain, the sitemap was exactly the same, the title/description were the same but the content was not fully similar (70% according to some web tools).

The also said that such subdomains would dilute the website authority and would make the main website less competitive.

Anyway, I'm curious to hear what you and others think about that.

Thanks again :)

[vaibhav45]

But tedster sites like expedia, booking.com or other big brands have site for each country with same content for eg abc.co.uk, abc.co.in, abc.com but google doesn't penalize them

[sleidia]

Have you checked example.co.uk? ;)

Also, a quick check of booking.com show that countries/languages aren't targetted to subdomains.

[edited by: tedster at 4:49 pm (utc) on Apr 30, 2010]
[edit reason] use "example" for domain names please [/edit]

[tedster]

Reference this Google blog article: Demystifying the "duplicate content penalty" [googlewebmastercentral.blogspot.com]

By actually geo-targeting each subdomain, you effectively remove them from competing with each other in the search results. Google will not cluster the nearly duplicate URL together, in order to choose the most appropriate result for a given searcher. They can use your geo-targeting preference. So you are not trying to deceive - you are clarifying things.

Did you have a real concrete problem, or did you make changes based on reading abstract ideas from others?

[sleidia]

The article you're refering to is one of the articles that made me remove those subdomains.

Especially these bits of text : "Don't create multiple pages, subdomains, or domains with substantially duplicate content."

"But most site owners whom I hear worrying about duplicate content aren't talking about scraping or domain farms; they're talking about things like having multiple URLs on the same domain that point to the same content. "

We could try to interpret what "substantially" means but with 70% similarity, I didn't want to take that risk.

Also, there is no clear info from Google stating that geolocalization prevents duplicate content issues.

The things is webmasters are so afraid to get sandboxed and at the same time Google doesn't give too many details about its functionning.

So, in the end, there is no choice but goint the "paranoia route". :(

[TheMadScientist]

Did you read the entire post?

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results. If your site suffers from duplicate content issues, and you don't follow the advice listed above, we do a good job of choosing a version of the content to show in our search results.

Or this:
[google.com...]

Use top-level domains: To help us serve the most appropriate version of a document, use top-level domains whenever possible to handle country-specific content. We're more likely to know that http://www.example.de contains Germany-focused content, for instance, than http://www.example.com/de or http://de.example.com.

Your sub-domains are essentially the same as directories.

The only way I would even consider doing anything different than what you are doing is using tlds (or using one other method that's totally different than anyone else is using, but that's another thread).

It also says to not use different domains on the post you're citing because that would be 'the same content on different domains', so each to their own on this, but IMO you're not going to gain a thing by moving it, will lose a portion of inbound link weight through a redirect, and you're basing your decision on a post about what people should not do with EXACTLY THE SAME content on a website to try to artificially raise their page and link count...

So, in the end, there is no choice but goint the "paranoia route". :(

If you follow both of the preceding pieces of advice presented by Google to the letter because you think they could apply to you and want to go the 'paranoia route' the only choice is not to present language variations or geo targets, which IMO is beyond tinfoil hat... If you truly have unique content on the variations, which would include language variations.

IMO They allow geo targeting because not everyone can get the tlds they need and they're trying to show visitors the right site, and keep from penalizing sites for showing the information to visitors... If you have the exact same content on all your sub-domains, then it will still be the exact same content in different directories and you should just combine them and leave your target world-wide if this is the case, because any of the solutions are EXACLTY what they're telling you to not do.

There's not really a good 'creative artificial workaround' to try to manipulate your rankings in other countries if you've only got exactly (or essentially) the same content to show everyone... Your domain is 'world-wide' if it's the same content on all variations, whether you use sub-domains, sub-directories or separate tlds it will be duplication, unless it's different content.

If the content is the same IMO you should combine it and work on links from the country you're trying to rank in. If not IMO you should leave it as is and work on links from the countries you're trying to rank in.

Sorry for the semi-rant, but it's either unique or it's not...

[tedster]

The early language in that article is, I believe, aimed at what used to be a relatively common spam attempt - using a keyword as a hostname to serve identical content and doing this thousands of times.

There used to be a flood of this kind of spam, and often each subdomain held only an index page.

Also, it is common to retain some of the original language when creating pages targeted to a variant of the language. There is not THAT much difference between UK English, South African English, Australian English, etc.

[sleidia]

TheMadScientist, yes I read the entire post :)

When I read " unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results." I'm left wondering what what Google means by "manipulating search engine results". The term is way too vague, which is why I chose the safest route. I mean, isn't the creation of subdomains to rank higher in other countries a way to manipulate search engine results?

As for the duplicate content issue, I should have mentioned earlier that all the countries I targeted had either French or English as their first language (usa, uk, australia, canada, france, belgium, swiss, quebec ). As you can see, the UK site and the US site where bound to be very similar (except some variations like targeted currencies and cities listings).

"it's either unique or it's not.." easy to say ... but what about partially unique? What does Google do in that case? I've read so many articles pointing tot the fact that Google was very smart at identifyinf small portions of duplicated texts.

Lastly, maybe I suck at understanding English but ... what do you recommend in the end? I don't understand whether you recommend the use of subdomains or not ... kind of left totally confused ;)

[OnlineConnect]

sleidia

I think tedster said setup was fine and you should leave it alone. I think when your on a path you have no understanding of you should go back to the start. Reset everything as before you started and read more online. It doesn't sound like you ever had an issue.

[TheMadScientist]

I mean, isn't the creation of subdomains to rank higher in other countries a way to manipulate search engine results?

Isn't that the same thing you'd be doing by moving them to their own directories too?

That's part of what I'm saying with 'if it's unique it's unique'. If it's unique enough to warrant the need for it's own directory, outside of search engine rankings, then IMO it's unique enough to be on it's own sub-domain. If it's not unique enough for it's own sub-domain, then IMO it's not unique enough for it's own directory either, because as far as search engines go sub-domains are treated very much like directories.

IOW This:
contry.example.com

Is not much different than:
example.com/country

To Search Engines as far as I know they're treated about the same.

"it's either unique or it's not.." easy to say ... but what about partially unique?

If the pages present different content for visitors to the extent the majority of the content is unique for visitors from each country, then personally I would separate the content. (Either using a sub-domain or directory is recommended.) If the content is 'essentially the same' then I would think about combining the pages and showing the information to everyone.

As for the duplicate content issue, I should have mentioned earlier that all the countries I targeted had either French or English as their first language (usa, uk, australia, canada, france, belgium, swiss, quebec ). As you can see, the UK site and the US site where bound to be very similar (except some variations like targeted currencies and cities listings).

I would definitely split the languages up, but might show the language specific pages to visitors from each country speaking those languages... So, in your case, where you are worried about duplicate content, I might do this:

en.example.com -> usa, uk, australia, canada
fr.example.com -> france, quebec, belgium, swiss

From there, I might, depending on the uniqueness of the content, separate the countries into their own directories:

ADDED: I just looked closer at your post, and the following is probably what I would do, because IMO if you're displaying different cities based on country those would definitely make a page unique, and would IMO cause the need for each having it's own page or directory. It really depends on the situation, but it seems really silly to display cities in Australia to visitors from the UK if their looking for information about the area they're in, but without seeing and understanding the site it's really tough to say absolutely for sure.

en.example.com/usa
en.example.com/uk
en.example.com/australia
en.example.com/canada

fr.example.com/france
fr.example.com/quebec
fr.example.com/belgium
fr.example.com/swiss

Just to make sure you see the difference in what I'm saying the first example would only be one directory:

en.example.com/usa-uk-australia-canada/
fr.example.com/france-quebec-belgium-swiss/

That's probably not what I would call the directories, but it is how I would structure the sub-domains. One sub-domain with one directory containing all the English content, and one sub-domain with one directory containing all the French content. I would then probably combine the four separate countries pages on a single page using the above structure. So there would not be a separate page or directory for usa, uk, canada or austraila... They would all visit the same page of en.example.com to find the information they are looking for.