Forum Disaster Prevention & Recovery

A timely topic...

[rogerd]

Forums present a unique challenge for disaster prevention and recovery. They tend to be very busy, with pages and/or databases being updated continuously. Reverting to a previous day's backup would represent content loss and be irritating to members. At the same time, forums tend to be relatively low generators of revenue compared to, say, ecommerce sites of the same level of activity. (I.e., a forum with a hundred posts an hour almost certainly doesn't generate the same revenue as an ecommerce site taking a hundred orders an hour.) Therefore, disaster-related solutions have to be cost-effective, too.

What precautions do you take for your forum for a variety of common (say, hard drive failure) and not-so-common (e.g., lengthy datacenter outage) problems?

[philbish]

Very true! My store database is maybe 2MB and the forum database is 60+MB

Backup to CD as often as you can. I think a 1 day blip is not too bad. Much better than loosing everything.

[rogerd]

Wow, talk about prescient... On a very busy forum of mine, the primary forum hard drive just failed. It didn't go out completely, fortunately, and we were able to pull a fresh backup and restore it after the datacenter guys changed the hardware. It was an outage of an hour and a half, a bit too long, perhaps, but it all came back. It was also fortunate that the main database was on a different server, as the amount of data would have been much bigger on that server.

Currently, we're doing a daily full backup of the database to a backup server at the host. We periodically downloaded to a local hard drive.

Although a multi-day, total datacenter outage like Westhost's recent problem is unlikely, in the current scenario my "fresh" backup would be as unreachable as the forum servers. We're thinking of going to a local download at least once or twice a week.

This still doesn't offer anything like shorter-term failover, though.

[linear]

Components of the strategy:
1) cron job to take timely backups using mysqldump (mine run 3 times/day now). If you're into fine-tuning, usually not every table is necessary to pull as frequently as that (how often do you cahnge board config?)
2) second cron job on different host (and ideally different network altogether) to fetch those backup files. It's tempting to have the backup with you--but if you don't have incredible bandwidth that follows you everywhere, you're better off leaving it on a second hosting account.
3) periodic rehearsals of recovery--if you can recover to another host account from your backups, then you are in good shape (if not, then adjust your technique).

If you are going to skimp, then skimp on number 2. Most of the errors I've encountered were not things that took the whole disk with them, so a local dump would save the day. But man, it sure feels good knowing that if the whole box got blown away (apologies for the pun) you could set up from backups in a few hours time. The way to know is to rehearse it.

I've done recovery unrehearsed enough to appreciate the value of a well-thought-out plan, with notes, checklists and so on.