homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Duplicate content from browser language detection?

 8:47 pm on Sep 5, 2008 (gmt 0)

Hi - I have 5 sites:

www.example.com (launched April '08)
fr.example.com (launched June '08)
de.example.com (launched June '08)
jp.example.com (launched June '08)
sg.example.com (launched June '08)

Content is localized dynamically depending on the browser's language setting.

But our traffic for www.site.com collapsed in Google after the country domains launched.

Is it possible that Google bot doesn't take the browser language setting in to account and just sees 5 duplicate sites?

Is there some way to test this?

[edited by: tedster at 8:51 pm (utc) on Sep. 5, 2008]
[edit reason] switch to example.com [/edit]



 8:56 pm on Sep 5, 2008 (gmt 0)

Whatever is going on with your rankings, it is not because of duplicate content. Search engines index character strings, not meanings - so the content is most definitely not duplicate.

I would strongly suspect the type of content negotiation you are implementing.

1. What happens when googlebot requests a page?
2. Is there a natural click path to each language version, or is it all automatic redirects?
3. How much cross-linking is involved? Does each page link to its four counterparts?


 9:03 pm on Sep 5, 2008 (gmt 0)

Thanks Ted, my responses below:

1. Can this be viewed in Goo Analytics or in some other web log analysis?
2. There is a natural click path to fr.example.com from www.example.com (a prominent country flag)
3. Very little cross linking (apart from point 2) on the basis that english version and english version readers share so little in common with other languages.

I hope I have understood your questions. Thanks again and please feel free to ask more questions. I will respond immediately.

[edited by: tedster at 9:04 pm (utc) on Sep. 5, 2008]


 9:24 pm on Sep 5, 2008 (gmt 0)

Your server logs should tell you a more detailed story. And your tech team also should be able to tell you how the googlebot user agent will be handled - at least how they planned it to be handled. Googlebot it isn't a browser and it won't be responding to language negotiation. So you need to make sure it can get a response for your urls in that situation.

I don't use GA very much but the accounts that I can see do not have a search engine bots report. As far as I know, you need your server server logs to see that.

My general recommendation is not to use automated language detection and forced redirects. First, when I travel to other countries, that type of site drives me wild with frustration. I think you're much better off allowing users to get the exact url that they asked for, and making the language choices they can make very clear on the page.

The technical "trick" of automated language detection can definitely backfire in many ways.


 11:55 pm on Sep 5, 2008 (gmt 0)

thanks again tedster, very generous of you. my work is cut out for me.

cheers - J


 6:26 pm on Sep 15, 2008 (gmt 0)


Here are my findings for the interest of others who find this topic:

Googlebot (not being a browser) does not reliably understand/process "Content Negotiation" and therefore thinks all these sites are identical (English version), and therefore may be subject to a duplicate content penalty and at the very least fewer pages indexed by Google. Technically, googlebot crawls without a preferred language setting and ignores "Accept-Language" which causes only the English page to be served when a specific URL is requested by Googlebot.

Short term solution:
. publish this tag <HTML lang="XX"> as part of every site. Replace the XX with the appropriate language code (fr, de, es etc.)
. have your server send this header: Content-Language: XX (again, replace the XX with the appropriate language)

[edited by: tedster at 7:40 pm (utc) on Sep. 15, 2008]


 4:23 pm on Sep 16, 2008 (gmt 0)


a quick question..

you said "3. How much cross-linking is involved? Does each page link to its four counterparts?"..

What do you mean by "four counterparts"?


 5:48 pm on Sep 16, 2008 (gmt 0)

johoney has five sites, and each one is a translation into another language. So in any given language, every page has four counterparts. Cross-linking every page directly to its four translations has been known to get a multi-lingual site into ranking problems. That kind of deep, internal and sitewide crosslinking between domains apparently runs into some kind of filter, or at least it has historically.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved