homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Indexing lots of dynamic pages which only have short lifespan.
Should I let google index these pages..

 11:20 am on Nov 20, 2007 (gmt 0)

Hi everyone..

Does anyone have any opinions on whether to let google index lots of dynamic content pages which are only valid for a short time? If you look at jobsites for example - jobs are only "live" for weeks and maximum months. Would you let google index these pages?

It would be great in that:
a) Google would see constant new content, added every day
b) A large amount of extra content

Jobs would get removed and although link would also get removed, the page would still be in the index

In this sitation I'd be inclined to let google index all the pages - once the jobs are removed the link will be removed and the page will overtime get removed from the index, or moved to supplemental, but no real harm in that.

The potential benefit overrides the cost in my opinion here but why doesn't everyone do this then?



 9:37 pm on Nov 20, 2007 (gmt 0)

It seems like this is what auction sites do, right? As long as there's also a substantial amount of long-lived content on the domain, I don't know of a problem with letting Google sort out those quickly 404 or 410 urls the way they prefer. Maybe someone else has some hands-on experience, because I'm just talking "in theory" here.

[edited by: tedster at 12:40 am (utc) on Nov. 22, 2007]


 4:46 pm on Nov 21, 2007 (gmt 0)

We run a dozen or so jobs sites portals and optimise sites for a couple of recruitment agencies for the last 4 or 5 years and have never had issues with this sort of thing.

We deal with this in different ways and none has ever caused us ranking problem. Historicall we have just 404 ed( with a custom 404 ) expired jobs.

On one site we 301 the expired job to its catagory index page and on others we keep the job listing and modify the content of the page to say that the job is no longer available.

The later option is now our prefered method and we have implimented this on new sites. The other methods we still use on legacy sites which still rank well.


 5:28 pm on Nov 21, 2007 (gmt 0)

Well here's my take.

IF - when the page dies - you can 301 the url to a search results page that offers new versions of the same job, then I would do that. But I would make it very clear to the user that while the job they WERE looking for is no longer there, here is a list of other jobs that have a similar profile.

I would use #searchphrase in the 301 redirect, instead of a?search=searchphrase syntax to help the search results page display just relevent jobs as Google has recently been saying that they ignore everything after a #.

If - on the other hand - that is all very complicated to build, then your best bet is to modify your 404 page to something static that simply says the page/job you are looking for seems to no longer be available and then help the user get to the next best page on your site.

* Don't create a situation where Google or usesr cannot generate a 404
* Don't 301 expired jobs/pages to pages that don't explain what happened or that just send users to the home page without explanation.


 5:46 pm on Nov 21, 2007 (gmt 0)

Thanks for this guys, interesting to hear your thoughts on this - especially thanks to you Conor for your practical advice..

Receptional - what exactly do you mean by "Don't create a situation where Google or user cannot generate a 404 ". Do you think that putting a "this is no longer available" message at the top is not a good idea?

Ideally - having a "this is not available - but look at the stuff below" message with similar content would be great for users, but yep, could be interesting from a technical perspective.

Am just thinking about my personal experience, but I seem to be finding less and less 404 pages on Google these days.. are people using them less and less?


 8:36 pm on Nov 21, 2007 (gmt 0)

Yes, Google has improved in not showing 404 urls in their search results - and that's because they are using the 404 responses that they get during spidering, not because fewer servers are showing a 404. I agree that it's very important to make sure that urls you do no have, or no longer have, generate a 404 or 410.

My take is that trying to leverage expired information on a rapidly changing website can lead to trouble. In several ways, it could begin to parllel a spammer's footprint. So I would prefer to simply go 404 or 410 when the content is no longer available.


 8:43 pm on Nov 21, 2007 (gmt 0)

I wouldn't advise 301 an expired job page to another specific job page, even if they are similar - sending a user to the index page of the same jobs catagory is fairly obvious to a user and to to Google.

Adding 'This job is no longer available' and placing a link to 'similar jobs' is not technically diffficult in any type of scripting environment.

[edited by: Robert_Charlton at 9:10 pm (utc) on Nov. 21, 2007]


 12:27 am on Nov 22, 2007 (gmt 0)

Thanks guys -

Tedster - "My take is that trying to leverage expired information on a rapidly changing website can lead to trouble. In several ways, it could begin to parllel a spammer's footprint. So I would prefer to simply go 404 or 410 when the content is no longer available. "

I can definitely see why you say this and if you have an ever increasing amount of expired content this is not going to look good. But - if the expired content has the links to it removed when it has expired, gradually (or at some stage!) the URL will be removed from the google index - right? So although expired content will build up.. at a certain stage (I don't know - does Google remove URL's after 6 months roughly?!) it will reach a maximum level.. where although new expired content will be added to site, old expired info will also be being deleted.. cancelling each other out (maybe!)

What do you think?


 12:48 am on Nov 22, 2007 (gmt 0)

I know Google keeps urls on record "somewhere" for years and years - even if you can't see them anymore, not even in the Supplemental Index. My preference, in a case like this, would be to keep things clean, rather than building up a pile of 301 redirects. I'm not saying that another approach definitely won't work - it certainly could, especially for a while.

I'm pretty cautious now with manipulating many areas to gain a bit of extra ranking or traffic. I'm working with several domains that have been burned pretty badly and are only now gaining a foothold again. So that might make me more conservative than the average. I tend to avoid redirects as much as possible, especially when they are not pointing to a new location for the original content.


 9:08 am on Nov 22, 2007 (gmt 0)

Another feasible option would be to ensure some (or a huge amount) or expiring content does not get indexed by just using no follow and no index tags on those pages - but allow a small proportion of the expirable content to get indexed.. less chance of getting burned, not that many 301's or 404's but still potential slight benefit from having those pages in the index..


 10:49 am on Nov 22, 2007 (gmt 0)

I return a page with a 200 response code.

I explain to the customer that the "job" is no longer available and I give a list of suggested jobs and categories that may be of interest.

I don't think this is fooling Google or the end user. Personally I think using a 301 to send them to another page is a step up from cloaking. You’re second guessing what the user is looking for.


 12:12 pm on Nov 22, 2007 (gmt 0)

Over time that builds thousands of pages that are all essentially showing the same Duplicate Content boiler-plate message.

By all means serve that message to visitors, but do it with a 404 status code, not the 200.

The 301 is also a good choice. Google will drop the original URL from their index in a short while once they see that status. Before that happens the redirect still sends the visitor to useful related content.


 12:18 pm on Nov 22, 2007 (gmt 0)

I agree that the time issue could play a big part in this - if all links are removed, but page and content are still there how will google react to it... does it slowly marginalise the page over time and then remove it from the index or as suggested, does it keep them for a very long time, move them ot supplemental and even after that keep them, and have them playing a part in the algorithm and the overall ranking of the site?

If i put no index and no follow tags on most if not all of the pages there should be no reason I get penalised or am looking like I'm doing something wrong - is that right?


 3:27 pm on Nov 22, 2007 (gmt 0)

I think that '410 Gone' is a better response code that '404 Not Found'. You can serve up a custom 410 page which says the job is gone but offer a search form for similar jobs (I would not use 301's here).

You can then have another 404 custom page which is for random incorrect urls. 410 implies that you know the page used to be there but now it is gone. Google will probably not index the content of the 410 page so you should be fine.


 4:30 pm on Nov 22, 2007 (gmt 0)

I started by serving an expired ad page which at end lead to problems with users that would find the ad they had erased on Google and ask me to remove it.

I'm now using 301s and I don't think this is problematic. Pages that had no links pointing to them will not be crawled at some point. On the other hand I can take advantage of the few pages which had external links by 301ing to the correct category.


 5:35 pm on Nov 22, 2007 (gmt 0)

I'm starting to think of this from a business point of view - if for example a customer puts something on my website for x amount of time... keeping it there afterwards may not be such a good idea.

Ok great, I could get some SE benefit from lots more content, but it could annoy my customers, who could justifiably ask me to remove it, and it could also upset people viewing.. who would be seeing something that's no longer relevant or has expired.

This may be one of those cases where taking Search Engines out of the equation is the way forward!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved