Welcome to WebmasterWorld Guest from 220.127.116.11
Does anyone have any opinions on whether to let google index lots of dynamic content pages which are only valid for a short time? If you look at jobsites for example - jobs are only "live" for weeks and maximum months. Would you let google index these pages?
It would be great in that:
a) Google would see constant new content, added every day
b) A large amount of extra content
Jobs would get removed and although link would also get removed, the page would still be in the index
In this sitation I'd be inclined to let google index all the pages - once the jobs are removed the link will be removed and the page will overtime get removed from the index, or moved to supplemental, but no real harm in that.
The potential benefit overrides the cost in my opinion here but why doesn't everyone do this then?
[edited by: tedster at 12:40 am (utc) on Nov. 22, 2007]
We deal with this in different ways and none has ever caused us ranking problem. Historicall we have just 404 ed( with a custom 404 ) expired jobs.
On one site we 301 the expired job to its catagory index page and on others we keep the job listing and modify the content of the page to say that the job is no longer available.
The later option is now our prefered method and we have implimented this on new sites. The other methods we still use on legacy sites which still rank well.
IF - when the page dies - you can 301 the url to a search results page that offers new versions of the same job, then I would do that. But I would make it very clear to the user that while the job they WERE looking for is no longer there, here is a list of other jobs that have a similar profile.
I would use #searchphrase in the 301 redirect, instead of a?search=searchphrase syntax to help the search results page display just relevent jobs as Google has recently been saying that they ignore everything after a #.
If - on the other hand - that is all very complicated to build, then your best bet is to modify your 404 page to something static that simply says the page/job you are looking for seems to no longer be available and then help the user get to the next best page on your site.
* Don't create a situation where Google or usesr cannot generate a 404
* Don't 301 expired jobs/pages to pages that don't explain what happened or that just send users to the home page without explanation.
Receptional - what exactly do you mean by "Don't create a situation where Google or user cannot generate a 404 ". Do you think that putting a "this is no longer available" message at the top is not a good idea?
Ideally - having a "this is not available - but look at the stuff below" message with similar content would be great for users, but yep, could be interesting from a technical perspective.
Am just thinking about my personal experience, but I seem to be finding less and less 404 pages on Google these days.. are people using them less and less?
My take is that trying to leverage expired information on a rapidly changing website can lead to trouble. In several ways, it could begin to parllel a spammer's footprint. So I would prefer to simply go 404 or 410 when the content is no longer available.
Adding 'This job is no longer available' and placing a link to 'similar jobs' is not technically diffficult in any type of scripting environment.
[edited by: Robert_Charlton at 9:10 pm (utc) on Nov. 21, 2007]
Tedster - "My take is that trying to leverage expired information on a rapidly changing website can lead to trouble. In several ways, it could begin to parllel a spammer's footprint. So I would prefer to simply go 404 or 410 when the content is no longer available. "
I can definitely see why you say this and if you have an ever increasing amount of expired content this is not going to look good. But - if the expired content has the links to it removed when it has expired, gradually (or at some stage!) the URL will be removed from the google index - right? So although expired content will build up.. at a certain stage (I don't know - does Google remove URL's after 6 months roughly?!) it will reach a maximum level.. where although new expired content will be added to site, old expired info will also be being deleted.. cancelling each other out (maybe!)
What do you think?
I'm pretty cautious now with manipulating many areas to gain a bit of extra ranking or traffic. I'm working with several domains that have been burned pretty badly and are only now gaining a foothold again. So that might make me more conservative than the average. I tend to avoid redirects as much as possible, especially when they are not pointing to a new location for the original content.
I explain to the customer that the "job" is no longer available and I give a list of suggested jobs and categories that may be of interest.
I don't think this is fooling Google or the end user. Personally I think using a 301 to send them to another page is a step up from cloaking. You’re second guessing what the user is looking for.
By all means serve that message to visitors, but do it with a 404 status code, not the 200.
The 301 is also a good choice. Google will drop the original URL from their index in a short while once they see that status. Before that happens the redirect still sends the visitor to useful related content.
If i put no index and no follow tags on most if not all of the pages there should be no reason I get penalised or am looking like I'm doing something wrong - is that right?
You can then have another 404 custom page which is for random incorrect urls. 410 implies that you know the page used to be there but now it is gone. Google will probably not index the content of the 410 page so you should be fine.
I'm now using 301s and I don't think this is problematic. Pages that had no links pointing to them will not be crawled at some point. On the other hand I can take advantage of the few pages which had external links by 301ing to the correct category.
Ok great, I could get some SE benefit from lots more content, but it could annoy my customers, who could justifiably ask me to remove it, and it could also upset people viewing.. who would be seeing something that's no longer relevant or has expired.
This may be one of those cases where taking Search Engines out of the equation is the way forward!