Protect Your Business with Chaos Testing

You are migrating your database. This process took months; hundreds of hours went into building the compatibility pipeline; dozens of developers manually verified data; they worked weekends to meet the deadline. And then during migration, a power surge takes the system down...
No one could have foreseen it, but such a minor oversight can derail the entire project or, worse, cause irreparable damage to the database.
Humans are wired to be wary of the future; it's a biological tool designed to keep us safe in the face of new experiences. But even so, we rarely, if ever, tend to think of the worst-case scenario.

Unlikely Is Not Impossible

A friend of mine jokes: 'When you go live, the question isn't whether it will blow up. Rather, what color will the explosion be?' Jokes aside, there's a grain of truth in his words.
Development environments often try to recreate production environments as closely as possible, or vice versa. Unfortunately, computer systems are extremely unstable, so even the slightest discrepancy can have far-reaching consequences, and that's just the tip of the iceberg.

Systems fail, internet connections lag, servers crash, hard drives fail, data gets corrupted. All of this happens, and sometimes we have little control over when and how it occurs. Remember how in 2011, hundreds of users lost their data due to an AWS outage?
Could that happen again? No, but is it possible? Yes, and that's precisely why engineers always design fault-tolerant systems. If NASA hadn't developed a fault-tolerant system, Apollo 11 would have crashed on the Moon due to a computer error.
The Apollo 1202 error code meant the onboard computer was overloaded with tasks. Fortunately, NASA programmers anticipated this possibility and created a backup system that quickly rebooted the computer and freed memory for new calculations.

Minimizing Recovery Time

The story of the Moon landing is a prime example of what modern engineers call MTTR — minimizing the time to recover from a failure. If a disaster is unavoidable, our solution is to minimize the time needed to restore systems to working order.
Imagine you have two competing businesses: Corporation A experiences several system failures in a day, while Corporation B has a single failure. Without any additional information, everyone would want to be Corporation B.

But suppose Corporation A's MTTR (mean time to recovery) is about 20 seconds, and Corporation B's is 4 to 6 hours. If Corporation A had 20 failures in a day, total downtime would be 6 to 10 minutes. Suddenly, the frequency of system interruptions seems much less important.
How do we minimize recovery time? First and foremost, you need to purposefully take the system down. This is called a controlled failure. While it may seem counterintuitive, it makes a lot of sense when you think about it.

During a controlled failure, the date and time when the system will go down are announced, but the nature of the failure is not disclosed, so the team must diagnose the problem and restore the system as quickly as possible.
During this time, we monitor system data before, during, and after the failure. This is done not only to aid recovery but also to gather data for subsequent analysis and improvement.
Such exercises pave the way for new knowledge as the team discovers the impact of unexpected weaknesses in the system. It's a shift in perspective through shock therapy, when a supposedly fault-tolerant system suddenly reveals how fragile it is.
Thanks to the insights gained from these exercises, new procedures can be developed with a clearer understanding of their shortcomings. Although the exercise can be stressful, its results are more than worth it. And frankly, it can be one of the most intellectually challenging exercises for a development team.

Introduction to Chaos Testing

Controlled failure exercises are just the tip of the iceberg and a basic introduction to chaos testing/chaos engineering. At its core, chaos testing is simply about creating the ability to constantly, but randomly, cause failures in the production system.
Chaos engineering was a core strategy of streaming giant Netflix. The engineering team had a wide range of 'chaos monkeys' or potential failures that could strike at any moment, ranging from latency issues to a global Amazon Web Services outage.

These chaotic failures, in turn, push your team from defensive development to a more aggressive approach. More precisely, it's a method for building resilience in both the system and the team itself.
A resilient system has the flexibility to adapt to catastrophic circumstances, such as a streaming platform that reroutes its traffic when data latency suddenly changes.
A resilient team is flexible and open-minded, able to quickly adapt and develop new strategies when unforeseen problems arise. Resilient teams tend to view emergencies as opportunities for growth and adaptation, rather than fearing them or feeling stressed.

Resilience is something that can be built, both in terms of system dynamics and team dynamics, and chaos testing helps foster that mindset. It's like the fire drills we had as children. By simulating a crisis, we become accustomed to it and learn to stay calm when things go out of control.
It's important to note that this method is very demanding on developers and is not recommended for newly formed teams or small projects. It is suitable for projects with many moving parts where a single error can have large-scale consequences.
Chaos testing is strongly recommended as one of the best methods for improving fault tolerance and MTTR, and is used by software and engineering giants like IBM.

Provoking Chaos to Protect Your Business

Creating chaos might seem like an oxymoron, but the evidence is undeniable. Netflix is one of the most reliable systems on the planet, demonstrating just how effective chaos testing can be when done correctly.