caveman - 3:26 pm on Oct 3, 2005 (gmt 0)

pontifex, stever,

Your points about the need for G to fight spam are fully understood and appreciated. In fact, that is why I noted that I could understand if G were filtering the three subpages that expand upon the differences in the species, as long as they left the main page on the species in tact.

For a little more background, most of the sites we work on we do for ourselves, but I also do a little client work, and the site that first got my attention WRT this issue was a client site. It's as clean as clean gets, primarily a scientific site, and reasonably well known among those who care about the topic. If you are a scientist, you care about and minor differences within a genus/species, and for reasons not worth getting into here, the site was developed with different pages for each variation. It was not done for SEO reasons; it was done long before the site operators had any sense of SEO (which today for them is at best a necessary evil).

There was a time when G would show all four "bee" pages from this site (i.e., the main bee page and the three subpages). Late last year that began to change, and it became obvious over the last nine months that G was trying to sort out which "similar" pages to show, and which to dampen. I outlined above the general evolution of what they were doing, at least as far as I was able to work out.

One could look at this site, and argue the merits of the need for the three subpages. I believe they are warrented. Any scientist would concur. Some laypeople might not. In any case, the pages are not spam in the eyes of the site creators. The one main page and three subpages exist for a reason: to help the users of the site.

The problem, most probably, is that as pontifex and stever imply, there are certainly parallel sorts of examples out there where spammier sites are doing essentially the same thing, to get more pages, and capture more long tail searches. This phenomenon probably has it's root in the Florida Update, after which SEO'ers saw the benefits of having larger sites and more deep/specific pages...but took things to extremes in some cases.

G and the other SE's have an ever more difficult problem when it comes to controlling spam. I am sympathetic.

What I am not sympathetic to is overly harsh filtering of pages that are entirely legitimate. There was a time when G was finding a way to show the main "bees" page from the site I've alluded to, while substantially dampening the related subpages. This was not ideal for the site in question, because it caused a minor drop in traffic and put some searchers one page away from the ideal landing page. But I understood the issue G was dealing with (I think), and again, I was sympathetic.

The issue now is that, as has been true on other fronts as well, G has gone too far, taking out an increasing number of legitimate sites and pages in their effort to stem the rising tide of spam. From interaction I've had with the SE's, I'm not entirely sure that they always understand this. There is a tendency among the engineers to at times get so caught up in the fight against spam that they become almost cynical when told that too many innocents are being taken out as well.

So IMHO, the SE's need to hear it from us when too many excellent sites are being hurt. I do not believe that is G's intent.