SLIs / SLOs / SLAs - Service Level Objectives

 It is crucial to observe the system's health and which behaviours really matter.

There are three measurements that I'm going to deep dive in and they are 

• Service Level Indicators (SLI)
• Service Level l Objectives (SLO)
• Service Level Agreements (SLA)

The definitions of the SLI/SLO/SLA are mostly based on our experience, intuition and what users define as good measurement.

SLI describe basic properties of metrics that matter, SLO what values we want those metrics to have and SLA how we will react if we can't provide that expected service. Let us look at each of them and expand a bit more.

Indicators

Indicators are the first important measurement to define, they collect the quantitative measure of some aspect of the level of service that is provided. The are a common set of indicators that are selected which are the request latency, how long a it takes to return a response, the error rate, how many failures happen over a time, and system throughput which typically is measured in requests per second. Another important measurement is the availability of the system.

Objectives

SLO is a target value or a range of values for a service level that is measured by an SLI. It defines what is the threshold within the SLI must be to SLO to succeed. 

SLO is    SLI < target or 
SLO is    lower  bound < SLI < higher bound

To set appropriate targets or it is required to know the system behaviours, collect some historical data and decide with business and product owners what is the appropriate thresholds to adopt.

Choosing and publishing SLOs to external users sets expectations about how the service has to perform and it will reduce complaints.

Agreements

SLAs is the final pieces of the three, it is an explicit contract between users and system's owner that include consequences of meeting or missing the SLOs they contain. Consequences tend to be monetary or refund or compensation but it can be something else too. The question that SLA answer is "What happens if the SLO aren't met?"

Usually, SRE team support the definition and setting of SLI and SLO where the business and product are more suitable to define more appropriate SLAs.

Indicators - what the users care about

Systems can through a ton of metrics but not all of them are eligible to be indicators because not all can help to build proper SLOs. Also many indicators doesn't mean a good way to observe the systems since not everyone will be able to work out what these hypothetical indicators mean. Ideally four or five indicators will be just right. 

Indicators can be grouped by there are three major categories which are user-facing serving, storage, big data systems.

• User-facing serving systems, the metrics are the ones that customers care about, generally they are availability, latency and throughput
• Storage systems, often rely on latency, availability and durability
• Big Data systems which mostly use throughput and end-to-end latency

Indicators - How to aggregate

Most of the metrics are collected server-side because it is straightforward to instrument their applications and send metrics somewhere. However, some indicators, like the latency requires some client-side data to be most effective. 

How to aggregate the events is also a key part to provide the clearness and eof providing data to make the systems to present they real health. Data can be aggregated over a specific period, like 200 requests per minute but it is crucial to use the best mathematical function to show the state of the system. This can be misleading because if the average is used, it can cover up any request spikes which you may never be aware of. Functions like p95, p90 give a better overview how the systems is responding and allow to tune the configuration of the systems in a more resilient way.

Indicators - Standardize

I recommend that there is a common definition of SLIs across the company and teams therefore these values can be compared each other. To drive these conversations, a set of principals can be shared like

• Aggregation intervals
• Which requests are included
• Through which observability system the data is acquired
• Data access latency

Objectives - Defining objectives

First step to defining the objectives is to see what clients care about and step into their roles that figure out what they care about and not what can be measured on the system. 

SLOs must be very clear and understandable by people that have not developed the systems. They should define how they are measured and the conditions. An example is

• 99% of POST requests will complete in less than 400 ms

Moreover you need to use different data aggregation or different thresholds, it is far better to specify multiple SLOs.

Objectives - Set Expectations

Once the SLOs have been defined and proven they work internally, they can be published so that users know what to expect from the application.

There are two strategies to protect against not meeting SLOs

Keep a delta between internal and external SLOs so that you the company still have some room to respond to problems before thy become visible externally

Don't overachieve because users rely on what is the current state of the systems rather than on what it will become in the future. So if the actual systems perform better than its stated SLO, users will be happier than if the opposite happens. As consequence, uses will rely more on the actual system performance.