Indexing lots of dynamic pages which only have short lifespan.

It seems like this is what auction sites do, right? As long as there's also a substantial amount of long-lived content on the domain, I don't know of a problem with letting Google sort out those quickly 404 or 410 urls the way they prefer. Maybe someone else has some hands-on experience, because I'm just talking "in theory" here.

[edited by: tedster at 12:40 am (utc) on Nov. 22, 2007]

We run a dozen or so jobs sites portals and optimise sites for a couple of recruitment agencies for the last 4 or 5 years and have never had issues with this sort of thing.

We deal with this in different ways and none has ever caused us ranking problem. Historicall we have just 404 ed( with a custom 404 ) expired jobs.

On one site we 301 the expired job to its catagory index page and on others we keep the job listing and modify the content of the page to say that the job is no longer available.

The later option is now our prefered method and we have implimented this on new sites. The other methods we still use on legacy sites which still rank well.

Well here's my take.

IF - when the page dies - you can 301 the url to a search results page that offers new versions of the same job, then I would do that. But I would make it very clear to the user that while the job they WERE looking for is no longer there, here is a list of other jobs that have a similar profile.

I would use #searchphrase in the 301 redirect, instead of a?search=searchphrase syntax to help the search results page display just relevent jobs as Google has recently been saying that they ignore everything after a #.

If - on the other hand - that is all very complicated to build, then your best bet is to modify your 404 page to something static that simply says the page/job you are looking for seems to no longer be available and then help the user get to the next best page on your site.

* Don't create a situation where Google or usesr cannot generate a 404
* Don't 301 expired jobs/pages to pages that don't explain what happened or that just send users to the home page without explanation.

Thanks for this guys, interesting to hear your thoughts on this - especially thanks to you Conor for your practical advice..

Receptional - what exactly do you mean by "Don't create a situation where Google or user cannot generate a 404 ". Do you think that putting a "this is no longer available" message at the top is not a good idea?

Ideally - having a "this is not available - but look at the stuff below" message with similar content would be great for users, but yep, could be interesting from a technical perspective.

Am just thinking about my personal experience, but I seem to be finding less and less 404 pages on Google these days.. are people using them less and less?

Yes, Google has improved in not showing 404 urls in their search results - and that's because they are using the 404 responses that they get during spidering, not because fewer servers are showing a 404. I agree that it's very important to make sure that urls you do no have, or no longer have, generate a 404 or 410.

My take is that trying to leverage expired information on a rapidly changing website can lead to trouble. In several ways, it could begin to parllel a spammer's footprint. So I would prefer to simply go 404 or 410 when the content is no longer available.

I wouldn't advise 301 an expired job page to another specific job page, even if they are similar - sending a user to the index page of the same jobs catagory is fairly obvious to a user and to to Google.

Adding 'This job is no longer available' and placing a link to 'similar jobs' is not technically diffficult in any type of scripting environment.

[edited by: Robert_Charlton at 9:10 pm (utc) on Nov. 21, 2007]

Thanks guys -

Tedster - "My take is that trying to leverage expired information on a rapidly changing website can lead to trouble. In several ways, it could begin to parllel a spammer's footprint. So I would prefer to simply go 404 or 410 when the content is no longer available. "

I can definitely see why you say this and if you have an ever increasing amount of expired content this is not going to look good. But - if the expired content has the links to it removed when it has expired, gradually (or at some stage!) the URL will be removed from the google index - right? So although expired content will build up.. at a certain stage (I don't know - does Google remove URL's after 6 months roughly?!) it will reach a maximum level.. where although new expired content will be added to site, old expired info will also be being deleted.. cancelling each other out (maybe!)

What do you think?

I know Google keeps urls on record "somewhere" for years and years - even if you can't see them anymore, not even in the Supplemental Index. My preference, in a case like this, would be to keep things clean, rather than building up a pile of 301 redirects. I'm not saying that another approach definitely won't work - it certainly could, especially for a while.

I'm pretty cautious now with manipulating many areas to gain a bit of extra ranking or traffic. I'm working with several domains that have been burned pretty badly and are only now gaining a foothold again. So that might make me more conservative than the average. I tend to avoid redirects as much as possible, especially when they are not pointing to a new location for the original content.

Another feasible option would be to ensure some (or a huge amount) or expiring content does not get indexed by just using no follow and no index tags on those pages - but allow a small proportion of the expirable content to get indexed.. less chance of getting burned, not that many 301's or 404's but still potential slight benefit from having those pages in the index..

I return a page with a 200 response code.

I explain to the customer that the "job" is no longer available and I give a list of suggested jobs and categories that may be of interest.

I don't think this is fooling Google or the end user. Personally I think using a 301 to send them to another page is a step up from cloaking. You’re second guessing what the user is looking for.

Over time that builds thousands of pages that are all essentially showing the same Duplicate Content boiler-plate message.

By all means serve that message to visitors, but do it with a 404 status code, not the 200.

The 301 is also a good choice. Google will drop the original URL from their index in a short while once they see that status. Before that happens the redirect still sends the visitor to useful related content.

I agree that the time issue could play a big part in this - if all links are removed, but page and content are still there how will google react to it... does it slowly marginalise the page over time and then remove it from the index or as suggested, does it keep them for a very long time, move them ot supplemental and even after that keep them, and have them playing a part in the algorithm and the overall ranking of the site?

If i put no index and no follow tags on most if not all of the pages there should be no reason I get penalised or am looking like I'm doing something wrong - is that right?

I think that '410 Gone' is a better response code that '404 Not Found'. You can serve up a custom 410 page which says the job is gone but offer a search form for similar jobs (I would not use 301's here).

You can then have another 404 custom page which is for random incorrect urls. 410 implies that you know the page used to be there but now it is gone. Google will probably not index the content of the 410 page so you should be fine.

I started by serving an expired ad page which at end lead to problems with users that would find the ad they had erased on Google and ask me to remove it.

I'm now using 301s and I don't think this is problematic. Pages that had no links pointing to them will not be crawled at some point. On the other hand I can take advantage of the few pages which had external links by 301ing to the correct category.

I'm starting to think of this from a business point of view - if for example a customer puts something on my website for x amount of time... keeping it there afterwards may not be such a good idea.

Ok great, I could get some SE benefit from lots more content, but it could annoy my customers, who could justifiably ask me to remove it, and it could also upset people viewing.. who would be seeing something that's no longer relevant or has expired.

This may be one of those cases where taking Search Engines out of the equation is the way forward!