I do have a real problem with my largest website. It's basically a large multilingual site containing thousands of widgets.
Now my problem is: multilinguality. Imagine there are two languages. In one language there are 10-thousands of widgets available. The other offers only a few thousands.
Now when a user in language 2 searches for a certain widget he gets only 5 to 10 results. I show a link where he can include widgets in language 1 to increase the amount of results.
Google read the results of these links in his latest deepcrawl - and voila ... lots of pages got dropped.
(Original URL in Language 1)
www.widgets.cm/lang2/widgets/widget3000.htm (Lang 2)
(URL if displayed in Language 2 section)
The listing for Widget no. 3000 is available both in language1 and language2. All that changes is the language of the navigation and the additional functionality that is shown. The description of widget 3000 is still in language 1.
This setup is good for users which prefer language 2 because it makes it much easier for them to navigate.
Good for the users - but bad for the search engines.
Now, what happened:
Google read initially Widget 3000 in language 1. After a deepcrawl Google read Widget 3000 in language 2, found it to be dupe content and dropped both.
What I try to do:
Now each time somebody ( or Gbot ) is accessing Widget 3000 in language 2 i do a 301 to Widget 3000 in language 1.
Additional thoughts and my main question:
I do use Google sitemaps. For both language sections I do only list the widgets which are available in that language.
Now that I know that Google has read 10 thousands of widgets in the "wrong" language sections my target must be to deliver as much 301s as possible to fix that situation.
The question is: should I include my language2 urls which only point (301) to the according widget in language 1 in my sitemap files?
Any feedback is highly appreciated!
Thanks a lot and sorry for my rusty english - hope you understood what i meant.