Multiple Website Outage

[brotherhood of LAN]

[status.fastly.com...]

Affecting reddit, paypal, amazon, gov.uk, stackoverflow and many others.

[news.ycombinator.com...]

[engine]

I hadn't noticed, so I wonder if it has been fixed.

Added
I just stumbled across an error message on .gov.uk

[edited by: engine at 11:06 am (utc) on Jun 8, 2021]

[iamlost]

Monitoring
The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.
Posted 7 minutes ago. Jun 08, 2021 - 10:57 UTC

Identified
The issue has been identified and a fix is being implemented.
Posted 20 minutes ago. Jun 08, 2021 - 10:44 UTC

[Dimitri]

Fastly was down.

[cnet.com...]

[engine]

Its all been fixed now, according to fastly.

A fix was applied at 10:36 UTC. Customers may continue to experience decreased cache hit ratio and increased origin load as global services return.
Jun 8, 11:57 UTC

[engine]

I think we should all be concerned about this apparent weakness of the system.
More on this.

But a customer changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors", it said.

[bbc.co.uk...]

[iamlost]

The more lines of code the more opportunity for error aka bugs. The typical range for exploitable bugs in large open source projects is 0.05+/- per 1000 lines of code, which is ~1 per 20,000 loc.
Note: the 5-50 per 1000 sometimes seen is as initially written, prior to any review, test, etc. Hopefully not in anything live.
Note: for reference the Linux kernel currently has ~27.8 million loc so there may be a total (many already corrected) ~1390 exploitable bugs. Now multiple by how many softwares? It’s a numbers game - scale the platform scale the bugs.

That said, and no doubt Fastly is investigating means, fail overs and mitigations are supposed to limit such a catastrophic cascade.

They are far from the first and will not be the last. Enterprise by its nature means that when things go wrong they go wrong in a Titanic way.

Always worth a webdev doing an in-house check on their own code bases, their own potential failure bottlenecks, etc. and consider survival best practices. No point in being small, nimble, and adaptable if one goes down with the dinosaurs technical debt.