Forum Moderators: open
Often a user will post a link on the forum suggesting that someone use a certain search term with either of these site search engines.
.eg. mydomainname.com/search?q=keywords+like+this.php
My question is, should I block the results of these urls from the SEs with a robots.txt file? Or should I allow them to be spidered? I suppose I am asking whether the search results are unique content or duplicate content?
I had even considered setting up a kind of site map/internal directory of useful search terms using this technique. But, again, I wondered about the usefulness of this feature.
In some ways this is providing potentially useful information for the user (and also potentially new pages for search engines to spider), but at the same time the snippets of text in the search results are only drawn from already existing content.
Any thoughts/experiences anyone?
A few terms I NEVER would have thought of now generate nice traffic. One like this is my single most common SE phrase.
It's interesting, I get all sorts of traffic to my site from the 'unusual' titles of forum topics that users have started.
But I suppose I was concerned that the search approach would just generate duplicate issues. I wouldn't want 'real' static pages and forum threads to be dropped in favour of less useful search results.
The forum search engine results are a bit more sophisticated and display about ten lines of relevent text from the relevent threads with the search keyword highlighted.
So, whilst the results page is unique in itself, the content is drawn from already existing static pages.
This tactic was a very successful and well known technique that came to be seen as spam.
There was one very well known, very large site (think shopping search/comparison), that took every search they got and auto-generated a static page for it. It was killer in the SERP's but they eventually were outed for it and the SE's started looking for ways to put a stop to it, for exactly the reasons you site (no original content, just a technique to catch as many long tail search phrases as possible).
Plus, just generally, it's sorta bad form to create internal links that are just searches.
I believe that dup content filters is the way the SE's tackled this (but I'm really not at all sure about that). I do know that those kinds of pages started disappearing from the SERP's over time. As I recall, those kinds of pages eventually went Supp in G. In Yahoo, sites that did this seemed to get buried. Y's dup filters, IMHO, are in some ways very unforgiving. Been a while since I've looked to see how Y is handling stuff like this now.
Having said all that, it's clear to me that this tactic can and does still work, depending upon how it's implemented. I just wouldn't do it long term on a site I cared about.
What I would do, which relates to treeline's comments, is to constantly evaluate data from the log files, and use that to determine what pages the site should have, that it currently doesn't have. Then go ahead and build quality, useful pages. You won't catch as many long tail terms that was as with the auto-gen-search-result approach, but your site will be a whole lot safer IMO.
My 2 cents anyway. :-)
That's a spot-on perfect answer, of the kind that I was looking for. (Until someone comes along in a minute and gives an equally spot-on answer in the opposite direction. ;-)
I wasn't actually planning the tactic that you illustrated, much more of a lower key approach. But what you say makes sense.
