httpwebwitch - 9:00 pm on Sep 24, 2010 (gmt 0)

An automated system for verifying configuration values ended up causing much more damage than it fixed.

Well, then obviously those facebook folks are a bunch of drooling morons! ha ha ha

But seriously, their systems are so huge and complex, it impresses me that humans are capable of understanding it all. I have only been offered a brief glimpse into their persistent data storage system, and it's... gargantuan. It's a special kind of disaster when something incredibly complex starts melting down.

Good work getting it back up & running again, FB crew