HN Theater @HNTheaterMonth

> I think the biggest takeaway here is system-wide operations are bad for system-wide reliability.

Yes!

System-wide operations, whether they are human-driven operations ("ssh onto that box"), control-plane operations ("remove all the failed servers"), DI operations ("deploy the new code"), or even basic algorithmic things like replication ("put the same state onto all the servers") are the top causes of correlation that I've seen in the wild. Whether or not this matters to you depends a lot on what you're building, and how often you can tolerate failures. But if you're building something that needs high availability, durability, integrity, etc it's worth paying a huge amount of attention to the things that can introduce correlation in your systems.

If you're interested in reading more beyond what Joe (the OP) talks about some methods of avoiding those in the article (he's a colleague of mine at AWS):

* Our "Millions of Tiny Databases" paper goes into a lot of detail on another AWS take on reducing correlated failure (https://www.usenix.org/conference/nsdi20/presentation/brooke...). * Some AWS folks from the S3 team also touch on correlation in this talk: https://www.youtube.com/watch?v=DzRyrvUF-C0&t=2410s * I've written in the past about the role of software deployments in correlated failure (https://brooker.co.za/blog/2022/01/31/deployments.html), and about how to think about the role of redundancy (https://brooker.co.za/blog/2021/04/14/redundancy.html).

It's not really about the design of S3, but if you're interested in some of the philosophy and thinking behind S3 you might enjoy "Beyond eleven nines: Lessons from Amazon S3 culture of durability" https://www.youtube.com/watch?v=DzRyrvUF-C0