Facebook Service Unavailable - DNS failure message

[HuskyPup]

Ok, so where are you Facebook?

[SanDiego Art]

All those embedded like buttons are also showing up with that same error... Check your site if you implemented these for a big text block error message.

[Bewenched]

Guess someone didn't like all the publicity that they had on the news this morning.

[httpwebwitch]

My life is on hold

[maximillianos]

Yup my FB comments boxes are down.

[tedster]

Here are some tidbits from Twitter:

Tweetdeck: Facebook is currently suffering a major outage which is impacting TweetDeck FB columns too. We suggest removing FB accounts until fixed.

[twitter.com...]

Facebook: Facebook may be slow or unavailable for some people because of site issues. We're working to fix this quickly.

[twitter.com...]

Where's Facebook's version of the Fail Whale?

[LifeinAsia]

Seems to be alive now.

<unhold category="life" user="httpwebwitch" />

I like this tweet: tweet [twitter.com]:

BREAKING NEWS Facebook down. Worker productivity rises. US climbs out of recession.

[Bentler]

I saw this myself. Now I'm getting 503 status on a completely different site that hosts data. Could something bigger be up, like a dos attack/worm?

[robho]

Maybe the major DOS attack that brought down Nettica for hours yesterday was a practice run or related.

[tedster]

Here's a summary.

Facebook likely disappointed millions of bored office workers again on Thursday afternoon with a widespread outage and latency, a day after an outage shut down the site for hours...

According to AlertSite, a Website performance service and vendor, Facebook only had 38 percent availability with 60 second response times.

Meanwhile, until service was restored, frustrated Facebook users overwhelmingly turned to micro-blogging site Twitter to tweet their unhappiness -- a slight irony due to the fact that Twitter itself was the recipient of a massive cross-site scripting attack that bombarded users with pop-ups, rainbow tweets and #*$!ography just two days prior.

[crn.com...]

[engine]

More Details on Today's Outage [facebook.com]

The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition. An automated system for verifying configuration values ended up causing much more damage than it fixed.The intent of the automated system is to check for configuration values that are invalid in the cache and replace them with updated values from the persistent store. This works well for a transient problem with the cache, but it doesn’t work when the persistent store is invalid.



Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.

[maximillianos]

This same thing happens to my server on a much smaller scale. If my memcache were to suddenly think all cached objects needed to be replaced, my server crumbles to it's knees while the db gets hammered.

It is a risk/ tradeoff of highly cached system.

I can only imagine the issues they face at the scale they deal with. So much dynamic content. Crazy.

[ddogg]

The question is was your site traffic higher than normal yesterday due to FB not being available? Mine was, not sure it was due to this though. Facebook sucks all the air out of the room, I hate huge sites like this personally (unless of course I owned a site like this, then I wouldn't mind too much).

[maximillianos]

Interesting observation/theory ddogg. If it were down a whole day I bet we would see some different numbers.

[Seb7]

[bbc.co.uk ]

One of Facebook's senior engineers Robert Johnson apologised to everyone who couldn't log on.

In a statement on his blog he said: "The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition.

"An automated system [to fix the problem] ended up causing more damage than it fixed."

[httpwebwitch]

An automated system for verifying configuration values ended up causing much more damage than it fixed.

Well, then obviously those facebook folks are a bunch of drooling morons! ha ha ha

But seriously, their systems are so huge and complex, it impresses me that humans are capable of understanding it all. I have only been offered a brief glimpse into their persistent data storage system, and it's... gargantuan. It's a special kind of disaster when something incredibly complex starts melting down.

Good work getting it back up & running again, FB crew

[Sgt_Kickaxe]

I knew something was up, i clicked on a like button last night and the popup box was opening and closing itself over and over until I shut the browser.

[jeyKay]

BREAKING NEWS Facebook down. Worker productivity rises. US climbs out of recession.

lol

[tedster]

That senior engineer, Robert Johnson, wrote a longer article about the problem here: [facebook.com...] Essentially they had a problem that they could only fix by shutting down the site and then bringing it back online little by little. Sort of like rebooting your PC.