I have a site that shows dynamically generated content that becomes stale at a specified date in the future.
What I've been doing so far is when the page expires to redirect Googlebot (http - user agent) to the home page and to human users (coming as Google referals) show the original page with a warning that the content may be out of date. (From a bussiness perspective it's better to show humans the content rather than a 404)
I realize that this is not an optimal solution.
Would it be better to give crawlers 404 and users the page with warning, or when the content expires add <meta robots - noindex>?
Welcome to the board. I see content as some expensive good. Hard to throw away for both bots and visitors.
Here is the simple thing I do for expired content on some sites. I warn the visitor using text on a bigger than banner sized gif linking to fresher version. It only adds a line of code to expired pages, no one gets 404 and fresher content is easily found.
This requires unique URL for each documents and more efforts updating links to current versions. But it pays where I use it.
I heard Googlebot was following meta refresh, not sure...