pete - 3:34 pm on Aug 2, 2002 (gmt 0)

Great posts :)

Whilst I agree with Rich that there are a number of good reasons for displaying duplicate content, if the search engines didnt have a method of eliminating very similar pages, their results would be flooded with similar pages taking up top positions and a first page result set would be taken up by one network of very similar or identical pages.

At some stage, almost all of the search engines experienced this in their development. This was extremely prevalent in Excite and Infoseek (although very different engines) and Google, Alltheweb and Altavista all suffered with displaying multiple results of similar or identical pages. This definitely compromised their value offering and they quickly moved to reduce it with various mechanisms.

In the last six months, we have built and introduced 12 sites for clients (strong SEO focus). Small numbers by comparison to most of you. Almost all hit the mark after the usual lag (submissions to Yahoo!, ODP and Looskmart b4 pay for play), aggressive reciprocal linking and then big crawls and subsequent listings.

One missed the boat entirely. After seven months, not one page has been indexed by Google (Googlebot hits the site hard every month). The site does not differ from the other that we have built and importantly has a number of great links pointing to it (including all the directory listings necessary).

The client had 3 domain names and wanted us to alias 2 of them to point to the new one. In the first month, Googlebot ran through all 3 domains (by following links from prominent sites) and continued to do so in the next 3 months without listing any of the sites pages. We removed the aliasing in month 4 and have diligently been gaining reciprocal links from qualified sites.

I am convinced that we have been nailed by a duplicate content filter. Please shout if you feel otherwise and importantly, if you have suggestions of how to get out of this mess.