Welcome to WebmasterWorld Guest from 54.196.175.173

Forum Moderators: httpwebwitch & not2easy

Message Too Old, No Replies

Facebook Service Unavailable - DNS failure message

     

HuskyPup

8:00 pm on Sep 23, 2010 (gmt 0)



Ok, so where are you Facebook?

SanDiego Art

8:24 pm on Sep 23, 2010 (gmt 0)

10+ Year Member



All those embedded like buttons are also showing up with that same error... Check your site if you implemented these for a big text block error message.

Bewenched

8:28 pm on Sep 23, 2010 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Guess someone didn't like all the publicity that they had on the news this morning.

httpwebwitch

9:01 pm on Sep 23, 2010 (gmt 0)

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member



My life is on hold

maximillianos

9:02 pm on Sep 23, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yup my FB comments boxes are down.

tedster

9:06 pm on Sep 23, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Here are some tidbits from Twitter:

Tweetdeck: Facebook is currently suffering a major outage which is impacting TweetDeck FB columns too. We suggest removing FB accounts until fixed.

[twitter.com...]

Facebook: Facebook may be slow or unavailable for some people because of site issues. We're working to fix this quickly.

[twitter.com...]

Where's Facebook's version of the Fail Whale?

LifeinAsia

9:16 pm on Sep 23, 2010 (gmt 0)

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Seems to be alive now.

<unhold category="life" user="httpwebwitch" />

I like this tweet: tweet [twitter.com]:
BREAKING NEWS Facebook down. Worker productivity rises. US climbs out of recession.

Bentler

9:32 pm on Sep 23, 2010 (gmt 0)

10+ Year Member



I saw this myself. Now I'm getting 503 status on a completely different site that hosts data. Could something bigger be up, like a dos attack/worm?

robho

9:54 pm on Sep 23, 2010 (gmt 0)

10+ Year Member



Maybe the major DOS attack that brought down Nettica for hours yesterday was a practice run or related.

tedster

11:58 pm on Sep 23, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Here's a summary.

Facebook likely disappointed millions of bored office workers again on Thursday afternoon with a widespread outage and latency, a day after an outage shut down the site for hours...

According to AlertSite, a Website performance service and vendor, Facebook only had 38 percent availability with 60 second response times.

Meanwhile, until service was restored, frustrated Facebook users overwhelmingly turned to micro-blogging site Twitter to tweet their unhappiness -- a slight irony due to the fact that Twitter itself was the recipient of a massive cross-site scripting attack that bombarded users with pop-ups, rainbow tweets and #*$!ography just two days prior.

[crn.com...]

engine

12:03 pm on Sep 24, 2010 (gmt 0)

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



More Details on Today's Outage [facebook.com]
The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition. An automated system for verifying configuration values ended up causing much more damage than it fixed.The intent of the automated system is to check for configuration values that are invalid in the cache and replace them with updated values from the persistent store. This works well for a transient problem with the cache, but it doesn’t work when the persistent store is invalid.



Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.

maximillianos

1:45 pm on Sep 24, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This same thing happens to my server on a much smaller scale. If my memcache were to suddenly think all cached objects needed to be replaced, my server crumbles to it's knees while the db gets hammered.

It is a risk/ tradeoff of highly cached system.

I can only imagine the issues they face at the scale they deal with. So much dynamic content. Crazy.

ddogg

5:26 pm on Sep 24, 2010 (gmt 0)

10+ Year Member



The question is was your site traffic higher than normal yesterday due to FB not being available? Mine was, not sure it was due to this though. Facebook sucks all the air out of the room, I hate huge sites like this personally (unless of course I owned a site like this, then I wouldn't mind too much).

maximillianos

6:19 pm on Sep 24, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Interesting observation/theory ddogg. If it were down a whole day I bet we would see some different numbers.

Seb7

6:37 pm on Sep 24, 2010 (gmt 0)

5+ Year Member



[bbc.co.uk ]


One of Facebook's senior engineers Robert Johnson apologised to everyone who couldn't log on.

In a statement on his blog he said: "The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition.

"An automated system [to fix the problem] ended up causing more damage than it fixed."

httpwebwitch

9:00 pm on Sep 24, 2010 (gmt 0)

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member



An automated system for verifying configuration values ended up causing much more damage than it fixed.


Well, then obviously those facebook folks are a bunch of drooling morons! ha ha ha

But seriously, their systems are so huge and complex, it impresses me that humans are capable of understanding it all. I have only been offered a brief glimpse into their persistent data storage system, and it's... gargantuan. It's a special kind of disaster when something incredibly complex starts melting down.

Good work getting it back up & running again, FB crew

Sgt_Kickaxe

9:43 pm on Sep 24, 2010 (gmt 0)

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member



I knew something was up, i clicked on a like button last night and the popup box was opening and closing itself over and over until I shut the browser.

jeyKay

6:49 pm on Sep 25, 2010 (gmt 0)

5+ Year Member



BREAKING NEWS Facebook down. Worker productivity rises. US climbs out of recession.


lol

tedster

3:53 am on Sep 27, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



That senior engineer, Robert Johnson, wrote a longer article about the problem here: [facebook.com...] Essentially they had a problem that they could only fix by shutting down the site and then bringing it back online little by little. Sort of like rebooting your PC.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month