killroy - 8:31 pm on Apr 16, 2003 (gmt 0)

I have a large directory, that's been established for many years, and is the largest of its kind in this country. We provide information to 1000s of visitors every day. It's a messy legacy site that's evolved, rather then beeing designed, you know the kind.

Until recently I've never even thought about SEO and our site has done funny things with Google. Somebody (in a great flustered panic) pointed out that I was in serious danger of duplicate penalty. Basically the site has over 200000 pages in Googles index despite only showing about 3000 category listings and around 30000 content pages. The reason was following:

Each content page was linked to like this:
/category/sub-category/contentpage

but one page could be in several categories so
/category1/sub-category/contentpage
/category1/sub-category2/contentpage
/category2/contentpage

were the same page. I used the path to provide a return link on the contentpage which made navigation easier. Also many "related categories" and "other interesting related content" style links were loved by google. For some results we were dominating several pages of the search results.

Having been pointed out to the duplication penalty, and also noticing that of course each page had its own PR and a page that might have potentially PR 5 or PR 6 might in fact have 3 times PR 3 and a few PR 4 instead, I have now changed all links to point to:
/content/contentid
Of course that is not as pretty and doesn't make any directory/navigational sense, but I don't know how I could maintain my neat directories and still keep Google happy.

Would it be better to 301 redirect like this:
/category1/contentpage1 -> /category2/contentpage1
I wonder. But of course a 301 changes the browser display and would confuse the visitors.

How can I handle this case best where Googles requirements actually prevent me from delivering a better user experience? (user readable URLS and so on)

In fact we have often discussed "theme pyramids" and hierarchical directory structures, but what if the leaf nodes can reside in several branches?

Thanks for your insightful comments.

PS: Note this is a directory, it does not sell anything to its visitors.

SN