Reliability comes at a cost: maximising stability limits how fast new features can be developed and how quickly products can be delivered to user.
SRE seeks to balance the risk of unavailability with the goals to innovate and efficient services with features and service and perofmance
Unreilable systems can quickly erode users' confidence.
The cost has two dimensions
- The cost of redundance machine to allow unforseen maintenance or offline routine
- The opportunity cost is when an organization allocates engineering resources to build reliable systems
SRE team manage service reliability by managing risk. The SRE identify the appropriate level of tolerance for the serivces they run. They strive to make the service reliable enough but not more reliable than it needs to be.
Measuring Service Risk
The first step in measuring risks is to identify an objective metric that represents the property of the system that we aim to optimize. We can assess our current system performance and track improvements or degradations over time by setting a target. A system can be impaired by many events, we will treat all of them as unplanned downtime. The most straightforward way to figure out the acceptable level of unplanned downtime is to capture the desired level of service availability usually expressed in terms of "nines" like 99,9%. However, the uptime concept can have multiple meanings and for most systems, usually the frontend and backend systems, the definition of availability is aligned to the request success rate. Example: A system that successfully serves 1M requests in a day with a daily availability target of 99,99% can serve up to 100 errors and still hits the target. Using the RSR allows to the wide utilization of this objective metric and use it on multiple-layer systems.
Risk Tolerance of Services
To identify the risk tolerance of a service, SRE must collaborate with the product owners to turn a set of business goals into explicit objectives. There are many factors to consider when assessing the risk tolerance with the product owners
- Target level of availability
- Types of failures
- Cost
- Other service metrics
Also, the requirements for building and running infrastructure components differ from the requirements for consumer products in a number of ways.
Motivation and forming error budgets
The goals of developer teams and SRE are evaluated using different metrics. Product development performance is largely evaluated on product velocity which creates an incentive to push new code as quickly as possible. Meanwhile, the SRE performance is evaluated on systems reliability and stability, which implies an incentive to push back against an high rate of changes. These two points of views imply that somewhere the goals of the two teams has to meet and the error budget concept is the key. When the two teams jointly define a error budget, they have a clear object definition to set the product velocity vs realiability. As solong as the uptime measured is above the SLO, which means there are still error budgets to consume, new releases can be pushed. The main benefits of an error budget is that there is a right balance between innovation and reliability and teams can not complain since they signed a mutual agreement. If SLO is broken often, the speed of release has to decrease or even stopped, or if there is no break of the SLO at all, the speed can even increase.