oddsod - 10:45 am on Feb 23, 2010 (gmt 0)

Now you see how much that nightly backup service is actually worth during a mass outage....having all your eggs in a single host basket will never do anything but massively fail in the end.

I don't have all my eggs in one hosting basket. I take great care to use multiple hosts. Even with the site in my signature it's only the forum that is with Westhost, the rest of the site is elsewhere.

One solution doesn't work for all webmasters. Some may have just one or two large sites in which case it's easier to create and manage emergency plans. Others like me earn from a portfolio of sites and deal with risk (and for IP reasons) by spreading the hosting around. Even among the large site brigade, the webmaster earning $10K a month from his site may choose a $99 a month solution that backs up evey night while the guy making just $500 may choose to do manual backups once a month. And some of them may not need any backups at all. One of my best sites is completely static and the copy on my hard disk (and my hard disk backups) is enough security for me.

Different strokes for different folks.

according to forum posts by official Westhost employees on the Westhost forums the test of the fire suppression system was conducted by Consonus

According to phone calls with Consonus, there's no way Inergen can cause these problems, it's been tested in DCs before. What they do accept is that if the environment is not up to standard - loose bits and pieces lying about the DC - the high pressure released by the fire suppresant can cause objects to move about and cause damage.

This kind of fire suppressing system is present in many--if not most--data centers. That it happened at Westhost seems just to be a case of bad luck. It could--and can--happen in other data centers of other providers.

This week of downtime is not bad luck, it's bad management. They go to great lengths on their site to explain how safe their DCs are - earthquake protection, power surges protection, power cuts plans, generators, several days worth of diesel in underground tanks etc. etc. But the realities emerging are that their protection and their recovery "plans" have as many holes as a roll of swiss cheese.

Everybody takes ultimate responsibility for their own data, yes, but when a host has screwed up, the host has screwed up. Screwed up not just on planning for the contingency but on the subsequent recovery. Storing backup servers in the same DC is but one example of their ineptitude. Westhost owners editing the Wikipedia entry in the last few hours to try and cover up their mismanagement doesn't do them any credit either.