Step 1. Know your risks
Many people do not realize how much risk is inherent in their applications. Much of this risk is in the form of technical debt in the code, but some of it is based on known decisions about how the system should operate that imply unknown outcomes.
Donald Rumsfeld, the previous United States Secretary of State, famously said that there are “known knowns” and there are “known unknowns,” but that the problems to be concerned about are the “unknown unknowns”—the problems that we don’t know about.
Risk management is about removing the unknowns and making their knowns. In modern applications, risk management is about identifying areas of concern, labeling them, quantifying them, and prioritizing them. Then, addressing the risks that have the highest impact on our business.
To do this, each development team for each service in your application should create and maintain a risk matrix. A risk matrix is a spreadsheet containing as many issues and potential issues as possible. It’s a brainstorm by everyone with a stake in the service to identify as many risks as possible. Then, for each risk, they are assigned two numbers:
- A severity specifies how serious of a problem it would be for our business if this risk were to happen.
- A likelihood, which specifies how likely this risk is to occur.
A risk can have a high severity but a low likelihood, meaning that it isn’t likely to happen, but the impact would be significant if it does. It can have a high likelihood but a low severity, which means the risk is more than likely to occur but won’t be a serious problem.
The most concerning risks are the ones that have a high likelihood and high severity. They pose severe problems to our business and are likely to happen. These are the highest impact risks.
The risk matrix provides a model for each team to prioritize their operational workload to understand what is important to work on and what is not important. Done correctly and consistently, it can prioritize risks across teams and allow management to allocate resources to the greatest issues.
Risk matrices give visibility and prioritization to technical debt and pending problems. They are a great communications tool between development teams and management.
Effective use of risk matrices will help reduce availability issues in your application.
Step 2. Watch your software
Understanding what your software and your operational infrastructure are doing at any given time is critical to maintaining high availability. Application and infrastructure analytics can give you insight into how your application is performing, allowing you to tune and optimize your operational environment, detect and resolve live operational problems, and understand who is using your software and how they are using it.
Used and set up properly, analytics can give early indications of pending availability problems, allowing you to fix an application or operational issue before it becomes an availability problem.
There are many free and paid systems and services that provide application and infrastructure metrics and analytics. All of them have advantages and disadvantages. Free systems are valuable for those who want to build and maintain their own systems and even customize them to fit their particular needs. Paid systems can offer a more hands-off experience but often require a significant financial investment. More modern paid systems even offer AI systems that analyze your application performance and give you early indicators of problems that you may not even notice among the depths of data available.
A full system to analyze your software provides the ability to:
- Monitor your system continues to know how it is working.
- Examine changes in performance around deployments to see if a deployment may have introduced a problem or to verify a problem has been resolved.
- Inform you via notifications when anomalies of various sizes or shapes are detected, allowing you to look at deeper data to determine what might have gone wrong.
- Assist you in resolving an ongoing incident, using data that can help understand why a particular problem occurs.
Analytics is also a great way to monitor service-level agreements (SLAs). This includes public SLAs (those visible to customers) and internal SLAs (those that describe commitments between and among internal services). Analytics is a great tool for inter-team communications.
Step 3. Reduce your technical debt
Once you have analytics in place and have identified your technical debt and other problems via your risk matrix and other tools, you need to evaluate and reduce your highest-impact problems. Knowing what your problems are is great, but it doesn’t help if you don’t reduce those problems.
If you have a high-severity, high-likelihood risk on your matrix driving availability issues, it must be fixed. But fixing it doesn’t necessarily mean rewriting to remove the risk. You can resolve the availability issue by reducing either the severity of the likelihood of the risk.
In other words, if you can’t easily remove an issue that’s causing you problems, then either make the issue happen less often—so that it’s not a frequent source of concern—or reduce the impact of the problem when it does occur by reducing the severity. Either way, the result is that the problem is no longer a major driver. It may still be a recognized risk, but the reduced frequency or reduced impact makes it no longer a critical concern.
Having a regular focus on technical debt helps keep availability in line. But be careful. You aren’t looking for perfection. Your goal should never be to remove all technical debt and hence remove all risk. Unless you are building the control software for an airplane, rocket, or similar system, you need to balance effort with the impact of the problem. Focusing on reducing technical debt too far may indicate that you are spending too much time focusing on “perfecting” software at the cost of some other business opportunity.
Step 4. Automate recovery as much as possible
When an incident does occur, how long it takes to recover can have a huge impact on your overall application availability. It’s important to recover fast. It’s also important to correctly diagnose the problem and take steps to ensure it doesn’t occur again.
When an availability incident happens, the response generally involves the following steps:
- You notice that a problem occurs (either you detect the problem or a customer reports the problem).
- You analyze what’s causing the problem.
- You roll out remediation to reduce or eliminate the problem.
- You implement a permanent fix, if necessary.
- You hold a post mortem on the episode.
This same sequence of events occurs every time there is an event. The problem is this process takes time. When the problem occurs, or when it is first noticed, and when remediation is put in place to remove the problem is called the meantime to repair (MTTR). The longer your MTTR, the lower your availability. Because humans are involved in diagnosing and fixing the problem, your MTTR can be quite long, impacting customer satisfaction.
However, sometimes you are aware of certain types of problems that can occur, and the process to fix the problem can be quiet and automated. By automating the repair of these types of problems, you can dramatically improve your MTTR.
A classic example of an automatable repair is when a computer instance goes offline. This can happen due to a software problem, a network problem, or another cause. But monitoring software can detect when the instance stops responding, and the instance can be immediately rebooted. Or, in the cloud, the instance can be terminated and replaced with a new instance. This can occur automatically. Because a human doesn’t have to be involved, your MTTR for this class of problem can be reduced, which can improve your availability markedly.
Step 5. Try and break things regularly
The best way to keep your application operating is to try and break it regularly.
Yes, that’s right. You heard me correctly.
The operators of the biggest applications in the world regularly test their resilience to problems by trying to break their application regularly.
The idea is this: Your software will fail. But do you want it to fail in the middle of the night or at a critical time operationally? Or would you rather have it fail at a more opportune time, with your engineers looking on and ready to detect and fix the problem quicker?
In the first case, you provide a bad experience and potentially long-lasting damage to your customers while trying to figure out what’s wrong with the application. In the second case, you know what caused the problem (you caused it) and quickly fix it. In either case, you gain valuable experience on how your application operates. Your learnings are the same, but the costs of the lessons are far less.
There are two common ways to accomplish this production operation testing. The first is called game days. Game days are scheduled times when you inject specific failures into your operational infrastructure to see how the problem manifests and how quickly you can detect and fix the problem. A common game-day test scenario, for example, is to bring down an entire data center to see if your application can failover to a backup data center.
The second common method of production operation testing is called chaos testing. Chaos testing involves operating a software system that randomly and unpredictably breaks parts of your system regularly. This might involve crashing a server, breaking a network link, or taking a load balancer offline. Chaos testing is a great way to test automated recovery mechanisms and prove the safety and efficacy of your recovery processes.
In either case, the goal is to identify problems in a controlled manner, learn from the errors, and improve the quality of your application to self-repair from these failures. The twin goals of both approaches are to improve your operational reliability and improve your application availability.
Improve processes, improve availability
Improving application availability is not about striving for perfection or eliminating every risk. It is much more about improving your operational processes: working to reduce the severity and likelihood of problems, closely monitoring applications and infrastructure, keeping technical debt in check, automating recovering mechanisms, and regularly putting those recovery mechanisms to the test. Follow these steps, and your application availability will be markedly improved, your customers will be happier, and happier customers will mean more business for your company.
You must be logged in to post a comment.