Forum Moderators: Robert Charlton & goodroi
This question is regard to one of my websites which is 11 years old. Part of the webiste is a directory of websites created by a program called Hayperseek. The program creates can create either static or dynamic pages but I have chosen to create static pages for the past several years.
The program also creates the directory in two possible ways
1) All the pages of one directory or 2) separate directories for each category and subcategory. I had the site built both ways.
About two years ago, i began to notice that Google started giving good SERP to the directory. It always indexed the pages in the single directory and mostly ignored the duplicate pages in the multiple directory. On the other hand, Yahoo liked the multiple directories.
Occasionally I would switch back and forth between the two modes so that the site would appear to be build one way or the other.
About 10 months ago, i started creating a sitemap with Gsitecrawler, a program that creates a sitemap and uploads it to your site. I have submitted updated sitemaps since I started using the Google webmaster program.
My SERP improved gradually until last November 15th or so. I had a couple of server outaages and so when the results dropped, I attributed the drop to the server downtime.
But then I was on a roller-coaster. After about a month, my SERP shot up to the heights they had been at before the drop. Then they plunged again a week or so later.
I read about duplicate content. The first thing I thought of was the two directories. So I deleted the multiple directories and wrote permanent redirects from those pages to the corresponding pages in the single directory.
That was over a 1000 redirects. Its hard to tell what the results were because fluctuations between almost all time highs and lows kept happening.
I Begin to analyze some of the reports on the Google Webmaster area for answers. i noticed that it was showing an increasing number of pages which ended with his m/cgi-bin/search/hyperseek.cgi?ID=#*$!#*$!xx where #*$!#*$!xx are numbers.
I did not understand why the Gsitecrawler was restricting some of these and went to their help forum. Then I started looking at these pages in my site:domain.com results and noticed that these pages were actually other websites. The hyperseek had disguised redirects by creating these pages for each link.
So I thought that maybe these pages are considered duplicate content. I wrote a new robot.txt that restricted Google for the cgi-bin/search/ directory and made sitemaps that matched.
Its been over a week and things have not improved. A lot of what i have done is guesswork so what I want to ask is if anyone knows whether each of the two types of pages that I thought might be duplicate content are actually considered duplicate content and frowned on by Google?
