And welcome to the third video in our series

on SRE and DevOps.

My name is Seth, and I'm a Developer Advocate

at Google focused on infrastructure and operations.

LIZ FONG-JONES: And I'm Liz, an SRE focused

on teaching Google Cloud customers how to build

and operate reliable services.

SETH VARGO: In the previous video,

we discussed the differences between SLIs, SLOs, and SLAs.

SLIs are quantitative measurements, like latency.

SLOs are the amount of time that an SLI

can be out of specification.

And SLAs are business agreements with explicit consequences

for failing to deliver service.

But what stops teams from breaking their agreed upon SLOs

and forcing SREs to work overtime?

It seems like a classic DevOps problem where

product teams want to ship new features,

but SREs need to maintain a reliability.

Is there anything in the SRE program

that can help with this classic problem?

LIZ FONG-JONES: That's what we use error budgets for.

SETH VARGO: Error budgets?

LIZ FONG-JONES: Well, before we talk about error budgets,

let's talk about risk and availability.

As I mentioned in the previous episode, trying to go for 100%

availability just isn't a good idea because it's expensive.

It's technically complex.

And in a lot of cases, it winds up being the case

that users don't even see the benefits of it because

of end reliability somewhere else in this system.

SETH VARGO: I see.

That makes a lot of sense.

Because if my cellular network is only 99% reliable,

but my service is 99.9% reliable,

my users are never going to experience that additional 0.9%

of reliability because they're cellular network

is likely to fail before my service does.

LIZ FONG-JONES: Yes.

That's exactly correct.

So while we want to reduce the risk of system failures,

we have to accept some degree of risk

in order to deliver these products and features.

SETH VARGO: But how do we determine

how much risk a service is willing to tolerate?

LIZ FONG-JONES: So that's a product decision.

So we have to work with the product's management team

to figure out what is our explicit goal

for the availability target of our service.

And there are many things to think about,

like how much is it going to cost to add extra fault

tolerance, or to add extra testing time,

or to reduce our frequency of pushes,

or to increase how long it takes for us to decide

that a release is good compared to the benefits to the user

of increased reliability.

SETH VARGO: I see.

So the acceptable risk of a system dictates the SLO,

and the SLO mathematically defines the error budget.

If the service incurs too much downtime,

we have to reduce the risk to remain within the SLO, which

might mean halting deployments.

If service owners want to deliver

a lot of risky features, they have

to be willing to accept a much looser SLO.

Because if they were to choose a strict SLO,

they would quickly exceed their error budget, which

could halt future deployments.

LIZ FONG-JONES: Exactly.

So the main benefit of an error budget

is that it's a quantitative measurement that's

shared between the product and SRE teams, which

means that we can balance innovation and stability

to an appropriate level.

SETH VARGO: So as long as the SLOs are met,

releases can continue.

But how do we know if an SLO breach is about to occur?

LIZ FONG-JONES: So when we defined earlier the expectation

of how much uptime a service is going to have

and how we're going to measure it,

well, we need to actually concretely implement

that using a neutral third party,

like a monitoring system.

SETH VARGO: Well, and the metrics

on that monitoring system, those are the SLIs, right?

LIZ FONG-JONES: Exactly.

And the difference between the actual uptime

and the calculated target uptime from our SLO

is the budget of how much unavailability

that we can tolerate for the system

to be stable over the entire window of the SLO.

So we call this the error budget.

If your SLIs are failing all the time,

then you're going to be burning through your error budget.

And then eventually, you need to stop your feature releases

in order to focus instead on making reliability improvements

and restructuring your application so that it can

meet your SLOs in the future.

SETH VARGO: So who enforces those policies, though?

Because couldn't a product team just go over and break the SLO

and force the SREs to work overtime?

LIZ FONG-JONES: So this is why we

need to have executive buy-in for error budgets.

If the SRE teams don't have the ability

to enforce the error budgets, then the whole system

is going to break down.

So some teams just allow for a limited number

of tokens or golden bullets that you can hand out to a vice

president, for example.

So if a product team really wants

to get that critical feature out,

well, they're going to have to ask their vice

president for a one-time exception,

and they'll only get a certain number per year.

SETH VARGO: I see.

But what about things that are outside of the product team

that aren't necessarily buggy code for my developers,

like someone cuts an undersea cable,

or there's a catastrophic failure at a data center?

Those shouldn't impact my error budget.

It wasn't my fault.

LIZ FONG-JONES: So this is why it's

important to have error budgets from top to bottom

for everything in your stack.

That way you can figure out how much error budget you allocate

to your dependencies and how much error budget is reserved

for your developers to spend.

And this is another reason why targeting 100% availability

isn't realistic, because all of your dependencies

are not 100% available either.

SETH VARGO: That makes a lot of sense.

But what about other things like restarting a failed service

or other kind of manual tasks?

Are those considered part of the error budget?

LIZ FONG-JONES: Yeah.

So Seth, when you have to do manual action

to keep your system from failing, in the wait

before you actually do that manual action,

you'll start burning through your error budget.

But the actual act of doing that manual work,

we track that separately.

And that's a concept that we call toil.

So we'll talk about that more in detail in the next video.

SETH VARGO: Great.

So risk and error budgets are directly related

to many of the DevOps principles that we've

discussed in earlier episodes.

It clearly defines that accidents

are normal by quantifying accidents

and risk through error budgets.

It also enforces that change should be gradual

because a non-gradual change could quickly

burn through the error budget for a particular product

breaking the SLO and preventing further deployment

for the quarter or for the year.

This has really helped a lot.

I think it's really clear why we say that class SRE implements

DevOps.

LIZ FONG-JONES: Thanks, everyone, for watching.

Check the description below for links and more information.

Don't forget to subscribe to the channel

and stay tuned for our next video

where we talk about toil budgets.