Programmer is saying there is no dupe content penalty

I suspect he's trying to make his life easier

[SEOMike]

I'm working with a client who is moving a lot of their websites from a CMS to static hosting. The reasons for the move are... many. :) There wasn't much concern for dupe content problems on the sites before, (even though all the content was dupped) because the URLs from the CMS were unspiderable. Now the sites will have spider friendly URLs and I'm concerned about duplicate content issues. There are over 60 sites that will all have the exact same content for most of their pages with the exception of a few unique pages. All the sites will be on the same class C. I'm not concerned about it because I've advised them to structure the sites so I can use the robots.txt to keep spiders out of dupped content. This all sounds logical to me. The programmer is pushing back saying that duplicate content isn't an issue the client should be concerned about. I've sited articles and my expertise but he is convinced he's right. I don't get it. I think he just wants to take some programming shortcut to increase his margin that he can't take when it's done my way.

Any thoughts on this? I'm pretty sure I haven't missed the "Dupe content is ok" announcement. He's just so convinced that he's right... I wonder why.

[LifeinAsia]

He's just so convinced that he's right.

Then let him sign an agreement making him liable for any duplicate content penalty (and resulting loss of revenue by the company) if he's wrong. :)

However, it may be good to CYA and have the company sign a waiver saying that you advised against following the programmer's advice, but that they agree with the him, so they will not hold you responsible if/when they get hit with the dupe content penalty.

[Receptional Andy]

Your programmer probably just lacks wide enough experience. Duplicate content isn't always a search engine problem. You can have a site riddled with duplicate content but with little noticeable effect. But you can have a site with a relatively low volume of dupes that takes a serious hit.

It depends on how and what is duplicated if you ask me. The trick is figuring out which you have ;)

[jimbeetle]

There are over 60 sites that will all have the exact same content for most of their pages with the exception of a few unique pages.

I'm not one for bandying the word "penalties" about, I believe that in actuality they are relatively few and far between.

But in this case, when the SEs see 60 sites with the exact same content coming online I'd think something more than a simple duplicate content filter is bound to be tripped. You'd better convince the client to convince the programmer to do that bit of extra work.

[Demaestro]

Then let him sign an agreement making him liable for any duplicate content penalty (and resulting loss of revenue by the company) if he's wrong.

A good idea, but it would be impossible to measure loss of revenue or even if penalties were issued as a direct result of duplicate content.

Perhaps his hesitance is because he thinks he won't get any extra money from doing this work because the job was taking on a flat fee or something? Is he refusing to do it or are you just trying to see his side of it?

If he won't do it you will need to convince his boss that this is required for SEO and if they don't accept that then not much you can do but hope for a day when they come to you and you can "I told you so" them

[Receptional Andy]

I'd think something more than a simple duplicate content filter is bound to be tripped

Just to play devil's advocate: not necessarily. Lazy example: Wikipedia.

Not that I don't advocate fixing duplicate content: IMO it has more harmful effects than just damage to search engine performance, and quite often seems to arise due to programming oversights.

At the same time, some circumstances where duplicate content exists would take more time and resources to fix than potential benefits gained.

IMO it's good to try to keep the developer on-side too. You want them to make the changes you suggest, after all. They may even feel a little hurt and indignant at your suggestion: you're telling them how to do their job. Try to talk 'em round (real-world counter-examples can work wonders) ;)

[sonjay]

If you are indeed using robots.txt to keep the spiders out of the duped content, then what's the problem? I would tend to agree with your programmer in this case.

[phranque]

it's not really a penalty per se, it is more like a dilution of page rank.

[jimbeetle]

Lazy example: Wikipedia.

Yeah, there are a lot of sites using wiki content, but pages aren't exact dupes -- different headers, different nav, different footers...mix this around enough and you no longer have duplicate pages. Same wit different implementations of DMOZ.

Sixty sites with the "exact" same content -- yeah, assuming page structure is the same -- is bound to raise red flags somewhere along the line. Also assuming that since the same programmer is doing the work that the nav, thus URL structure is going to be identical -- that's just trouble. It was easily sussed out and penalized by the SEs years ago, I have no reason to believe they can't suss it out today.

[Receptional Andy]

but pages aren't exact dupes -- different headers, different nav, different footers...mix this around enough and you no longer have duplicate page

Personally I've found exact dupes to be less problematic on the whole (probably easier to interpret?), although I thought the OP meant 'the exact same content' in different templates.

It also depends on what the client is expecting: 60 sites all performing well for the same words and phrases is obviously out of the question ;)

[SEOMike]

If you are indeed using robots.txt to keep the spiders out of the duped content, then what's the problem?

Part of the problem is making him put the pages into individual directories. He wants them all to live in the root!

or are you just trying to see his side of it?

Just trying to figure out why he might hold his beliefs strongly enough to try going against someone with 11 years of SEO experience. I talked to the client last night and they have instructed the programmer to lay out the directory structure per my recommendation.

60 sites all performing well

Nah, these are franchise sites with geographical specific URLs. I do the SEO for their corporate site and am trying to position their franchise sites for future SEO by applying a set of best practices during design. It'll make my life easier later.

[jtara]

The programmer is pushing back saying that duplicate content isn't an issue the client should be concerned about

Have you advised the programmer that there is a duplicate-content specification that they need be concerned about?

Who is the programmer working for, BTW?

And why the need for a programmer in the first place, for a static site?

He wants them all to live in the root!

An awfully strange idea coming from a programmer. Must not be a very good or experienced one. Of course, don't tell that to your client, but be informed by it.

(OK, I re-read and answered my own question above - I guess the programmer works for the client)

Even back in the dinosaur days of the 70's, programmers were taught "structured programming", (since replaced with more impressive-sounding paradigms (twenty cents!) like Design Patterns...) which means simply breaking a problem up into more simple components, and making use of hierarchy. It's a fundamental Computer Science 101 kinda thing.

Most programmers would automatically extend this methodology to the layout of data - in this case, HTML pages.

So, I'm kinda surprised and shocked that a programmer would opt to "put all the HTML documents in the root".

It doesn't make sense on so many levels that it's scary.