Application Observability

Monitoring SLIs and SLOs

Zac Bergquist

Service Level Indicators (SLIs) and Service Level Objectives (SLOs) are two of the most important concepts in Site Reliability Engineering (SRE), and are key to establishing an observability culture. It all starts with identifying the right SLIs. Think about the key features and workflows that your users care most about, and choose a small number (think three to five per service) of SLIs that represent the availability of these features. For example, if you are running an eCommerce service you may choose to measure the percentage of “add to shopping cart” operations that succeed. Next, choose an appropriate objective for each of these indicators. Avoid spending too much time debating the initial SLO - your first attempt will almost certainly need adjustment. A good guideline is to start with an initial SLO much lower than you think you’ll need, and fine-tune this number as necessary.

Once you’ve identified the SLIs and SLOs for your service you need to start measuring them. SLIs are typically measured over a rolling window so that the effects of an availability-impacting incident “stay with you” for some amount of time, before rolling out of the window. This approach helps teams to adjust their behavior in order to avoid multiple incidents occurring in a short amount of time, which leads to unhappy users. Since many services have different usage and traffic patterns on weekends, we recommend a 28-day rolling window - which ensures you’ll always have four weekends in the window.

Measurement

In this guide we’ll cover two simple frameworks for structuring your metrics in a way that facilitates this type of monitoring. While these two approaches might not work for every scenario, we’ve found that the majority of SLIs can be measured in this way.

The first approach uses a single gauge metric. A gauge is a metric whose value can increase or decrease over time, but in this approach we limit the value of the gauge to 1 (last probe was successful) or 0 (last probe was unsuccessful). Prometheus’ up time series works in this way.

The second approach uses two counter metrics. A counter is a metric whose value only increases with time (think total number of page views or total number of error responses). In this approach, the first counter represents the total number of probe attempts, and the second counter represents the total number of successful (or unsuccessful) probes.

The examples that follow will use Tanzu Observability by Wavefront for the storage and visualization of this data, but the concepts can be applied to any system that can store and display time-series data.

Monitoring SLIs with a gauge metric

This approach exposes a single metric that is either 1 or 0 at all times. For example, a probe could periodically test an application feature and set the value of the metric to 1 if the operation succeeded, or 0 otherwise.

The raw metric may look like this:

In this example, the probe was successful for all times except for a 2 hour block on March 2, a 1 hour block on March 30, and a 10 minute block on April 1.

If our SLO is the percentage of attempts that succeeded over our time window, then we need to determine both the total number of attempts in that window and the number of them that were successful. For the total number of attempts, we simply count the number of samples in the window (all 0 and 1 values). Since the value of the metric is always 1 when the operation is successful, we can simply sum all the samples in the window to get a value that represents the number of probes that were successful.

We can use the mcount and msum moving window functions to build the queries. For a metric named gauge.up, we would have the following queries:

• Attempts: mcount(28d, ts("guage.up"))
• Successes: msum(28d, ts("gauge.up"))

With this in place, the current SLO attainment is simply the ratio of successful attempts to total attempts. We can also plot the target SLO to make it easier to gauge how we’re doing at a glance.

Here we can see the 2 hour outage on March 2 took our availability from 100% down to 99.7% (still above our 99.5% SLO). The impact of the outage remains for 28 days, and the error budget is recovered on March 30, bringing the availability over the window back to 100%. This is short-lived, as the subsequent outages on March 30 and April 1 consume additional error budget, which will be reclaimed 28 days in the future.

Monitoring SLIs with two counter metrics

An alternative way to measure this type of data is to use two separate (but related) counters. The first counter represents the total number of attempts, and is incremented every time a probe is performed. The second counter is the number of successful attempts and is incremented only when the probe succeeds. If all is going well, these two counters increase at the same rate. When the system is unhealthy and some operations are failing, the total attempts counter will increase faster than the successful attempts counter. The tricky thing about counters is that they map “wrap” or reset due to a process restart or the maximum integer value exceeded. For example, here’s a counter showing the total number of attempts for the same sample:

In this example, a counter reset occurred sometime around March 1. A naive query that simply looks at a counter value prior to March 1 and again after March 1 may think that the total number of attempts decreased. For this reason, it is important to use query functions that can handle counter resets. In Wavefront Query Language, the ratediff function can be used to find out how much a value has increased from one sample to the next, even in the face of counter resets. We combine this with the moving sum of those increases over our 28 day window to get the total number of attempts (or successes). For counters named probe.attempts.total and probe.successes.total we would have the following queries:

• Attempts: msum(28d, ratediff(ts("probe.attempts.total")))
• Successes: msum(28d, ratediff(ts("probe.successes.total")))

As you can see, this produces an availability graph with the same shape as the gauge example.