Onders - 6:16 pm on Jun 21, 2011 (gmt 0)
I'm currently looking into indexing a lot of unique content pages which have a high turnover. Every month we get a couple of hundred new ones, and a couple of hundred need to be archived / removed.
Wanted to share my experiences and what I've learnt so far, as well as see if you guys have any suggestions.
Some of our pages need to be archived after a month.. some longer. What I am going to do is set up an expiry meta so that Google knows how long the page should be live for.
When the page is archived on our site (should correspond to the meta), then I'm going to put a 410 on the page (I can't find a huge amount of information on implementing 410's but Google seems to suggest it for permanently removed content)
I hope to have the 410 page dynamically pull in data about the page that was archived so that it shows something relevant for the user (i.e. if the archived page was about blue widgets, the 410 will have something like dark blue widgets).
I'm not going to put all these in the htaccess as it would become huge, and I'm not going to submit removal requests to get the pages removed from the index.
I think I'm doing enough - and complying with the guidelines / suggestions... well, I hope I am!