Forum Moderators: not2easy

Message Too Old, No Replies

Facebook Outage Oct 4, 2021: Backbone Router Configuration Disrupted All Traffic

         

engine

10:54 am on Oct 5, 2021 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Facebook's outage of 4th October, 2021 [webmasterworld.com] was one of the most significant as it lasted for many hours, and resulted in Facebook, WhatsApp and Instagram becoming unavailable. The services eventually returned about six hours after they first fell over.

This outage to a major business has significant impacts to revenues and share prices, so there are lessons to learn for everyone, no matter the business size and scope.

Facebook has published an explanation which explains that configuration changes on its backbone routers which coordinate network traffic to its datacenters caused the interruption to the services, which caused a cascading effect.

Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.


[engineering.fb.com...]

Cloudflare has published an interesting assessment of outage, describing the issues, and talking about BGP (Border Gateway Protocol).
[blog.cloudflare.com...]

Sgt_Kickaxe

12:05 pm on Oct 5, 2021 (gmt 0)



If you, as a webmaster, are not removing any dependency on 3rd party data and scripts from your own site you're not thinking of your users first. Test your site(s) by installing an ad blocker on your browser, turning on "3rd party" blocking to see if your site(s) still look normal can be eye opening.

Most sites don't look right from the CSS being 3rd party but it was much worse yesterday.

engine

4:50 pm on Oct 5, 2021 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Incredibly, billions of $ wiped off his personal and business values.

I suspect MZ will be adding a second string to avoid this happening again.

tangor

8:11 pm on Oct 5, 2021 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Oddly ... a few of my buddies who actually us FB as originally marketed (family and friends) never noticed it was down near seven hours. They don't live there or do everything on FB. :)