SLOs, by the numbers

Most people can't feel the difference between 99.9% and 99.99%. On paper it's one more digit. In practice it's 43 minutes of downtime a month versus 4.

An SLO is a promise about how often a service does its job, for example 99.9% of requests succeed. What's left over is the error budget, and it's a tool as much as a limit. If you can tolerate 43 minutes of downtime a month, you can take more risks: deploy more often, and live with the occasional bad release and rollback. You don't need canary analysis and automated rollbacks on day one.

As those capabilities mature, you can raise the promise, because you have what it takes to keep it. This page turns those percentages into time, so you can see what each step up costs.

Allowed downtime

An SLO needs a window. Without one, every blip is an outage and the work becomes reactive: each one gets chased, and nothing else gets done. Look at the Day row: at 99.9%, a 90-second blip spends the whole day's budget. Over 30 days, the same blip barely shows. Start with 30 days, or two weeks for some workloads.

A month here is 30 days, a quarter 90 days and a year 365.

If you enter your traffic, the table also shows the budget as failed requests. Most SLOs are counted that way: 99.9% of requests succeed, not 99.9% of minutes.

Each nine is ten times smaller

Each extra nine divides the budget by ten. The bars below are to scale, over 30 days.

1. 99%7h 12m
   A bad deploy that nobody notices until the next morning.
2. 99.9%43m 12s
   One bad deploy, caught and rolled back.
3. 99.99%4m 19s
   Enough time to get paged and open your laptop, but not to fix anything.
4. 99.999%26s
   Less than a typical rollback. Only automation reacts in time.

Somewhere between three and four nines, the budget gets shorter than the time it takes a person to notice something is wrong. Past that point, each extra nine needs more than people on call. It needs automation: canary analysis that stops a bad release, and rollbacks that run on their own. It also needs architecture: the service runs across many clusters in separate failure domains, and a release reaches them one at a time, with baking time in between, so a bad change only hits part of the traffic. That's the work a looser SLO lets you put off, and the reason to raise the promise only once it's in place.

How many incidents can you afford?

An error budget gets spent in incidents. Each one lasts as long as it takes to notice the problem plus the time to fix it, and for most teams that's tens of minutes, not seconds. Put in your own numbers to see how many your SLO can absorb.

This assumes the service is fully down during an incident. A partial outage spends the budget more slowly: with 10% of requests failing, an incident costs a tenth of its length. That's what failure domains and staged rollouts buy you. A bad change that only reaches one cluster out of ten costs a tenth of the budget.

Dependencies multiply

A service can't be more available than what it depends on. If a request needs five services and each one meets 99.9%, their failures add up and the request succeeds about 99.5% of the time. Your 99.9% promise is broken before your own code has failed once.

Your dependencies spend your error budget too. Their SLOs belong in your plan, just like your own deploys.

Redundancy is how you get the nines back. With two replicas of a dependency, a request only fails when both are down at once. The calculator assumes two things you have to make true. The replicas fail independently: two copies behind the same switch, in the same region, or running the same bad release fail together, and count as one. That's what failure domains are for. And a broken replica gets out of the way quickly, which isn't always the case: see a slow replica is worse than a dead one below.

Latency and the tail

A latency SLO is a ratio too: "99% of requests complete in under 300ms". The slowest 1% is the error budget, spent the same way as downtime. What's different is that averages hide slow requests, fan-out multiplies them, and a single bad instance can create a lot of them.

Averages hide the tail

20,000 simulated requests. Most are fast, and 3% hit something slow, like a garbage collection pause, a cold cache or a retry.

Fan-out makes the tail common

A user request rarely touches a single backend. A search hits every shard of the index, and a page load makes dozens of API calls. The request is only as fast as the slowest of them, so a 1-in-100 event for each backend becomes the common case for the user.

This is why a service that fans out needs its backends to hold a much tighter tail than the user-facing SLO. If users should see 300ms at p99 and each request calls 100 backends, every backend has to answer within 300ms at p99.99.

A slow replica is worse than a dead one

Redundancy assumes a broken replica gets out of the way quickly. When an instance crashes, connections fail right away, clients retry elsewhere, and health checks take it out. When it hangs, it keeps accepting requests and holds each one until the timeout. Until something removes it, it gets its full share of the traffic.

The fixes are all about failing fast. Set timeouts from the backend's real latency, not a round 10 seconds. Use outlier detection: eject an instance whose latency or errors stand out from its peers, without waiting for it to fail a health check. And for the requests that are already slow, send a second copy to another replica and take whichever answers first: a hedged request.

What else to measure

Availability and latency cover most services that answer requests, but they're not the only things users notice. A good signal measures something a user would complain about. If nobody notices, it doesn't matter.

The user isn't always outside the team. A service has two: the people who consume it, and the people who operate it. Operators have their own SLOs. For a team that manages a fleet of machines, they could be how quickly a fault is detected, how quickly it's remediated, or what share of issues are fixed automatically in under 10 minutes.

Correctness is the hardest to measure, because you rarely know the right answer for a real request. Probes help: synthetic requests whose answer you know in advance.

Durability's nines aren't a budget. You can wait out an outage, but you can't wait out lost data: every failure is permanent. Eleven nines is a design target, met with copies, checksums and repair, not something you spend on deploys.

What changes with AI workloads

Serving and training models brings the same ideas to new places. Some signals split in two, some can't be read from a status code, agents multiply failures, and at cluster scale, uptime stops being the right question.

Latency becomes two numbers

A model streams its answer one token at a time. Users notice two things: how long until the first token appears, the time to first token (TTFT), and how fast the rest arrives, the time per output token (TPOT). The total time depends on how long the answer is, so one latency SLO can't cover it: a long answer is slow even on a healthy system. So each gets its own target, for example: 99% of requests see their first token in under 1s, and tokens arrive at least every 50ms.

TTFT includes the time a request waits in the queue and the time to process the prompt, so it's usually the first thing to degrade under load. Serving research measures goodput: the rate of requests that meet both their TTFT and TPOT targets, instead of raw tokens per second. And when you're out of capacity, a rejected request (an HTTP 429) is still a request the user didn't get. Decide up front whether it counts against your availability.

A 200 can still be the wrong answer

For a model, a successful response isn't necessarily a correct one. A new model version, a different quantization, a kernel change or new hardware can change the answers without a single error in the logs.

Quality needs its own signals, measured on purpose. Evals are the model's version of probes: a fixed set of prompts with known good answers, run continuously against production. Add a sample of real requests graded by people or by another model, and user signals like how often people regenerate an answer. They're noisier and slower than a status code, so they work better as a gate on every change than as a page in the middle of the night.

Agents multiply failures

An agent that takes 20 steps to finish a task is a request with 20 dependencies, and the math from dependencies multiply applies.

Checking a step's result and redoing it when it's wrong is the agent's version of redundancy. It works for the same reason replicas do, and it fails for the same reason: when the retry makes the same mistake, the two attempts weren't independent.

Training: uptime is the wrong question

A large training job is one program running on thousands of GPUs at once. When one GPU fails, the whole job stops, goes back to its last checkpoint, and restarts. At that scale, something is always failing, and the job has to be built for it. Meta reported 419 unexpected interruptions during a 54-day stretch of Llama 3 pre-training on a 16,000-GPU cluster, about one every three hours, and attributed about 78% of them to hardware. They still kept the cluster doing useful work more than 90% of the time.

So the signal isn't uptime. It's goodput, in a slightly different sense than for serving: the share of time the cluster makes progress. Each interruption costs the work since the last checkpoint, plus the time to restart.

The defaults roughly match the Llama 3 numbers above (one interruption every 6 years per GPU gives one every 3 hours across 16,384 GPUs); they're not Meta's actual settings. On average an interruption loses half a checkpoint interval of work. The model ignores the time it takes to write checkpoints. The default price is roughly what reserved H100 capacity costs in 2026; on-demand costs more. Set your own.

That's why, at this scale, the work goes into recovering quickly. Every minute shaved off a restart is a minute of the whole cluster: finding a bad GPU before it fails the job, checkpointing often and cheaply, keeping spare machines ready, and restarting in minutes rather than hours. It's the training version of failing fast.

Further reading

• Service Level Objectives, from Google's SRE book.
• Alerting on SLOs, from the SRE workbook: how to page on how fast the error budget burns.
• The Tail at Scale, by Jeffrey Dean and Luiz André Barroso: fan-out, tail latency and hedged requests.
• DistServe: goodput for LLM serving, with TTFT and TPOT targets.
• The Llama 3 Herd of Models, section 3.3.4: what training on 16,000 GPUs looks like day to day.

← All notes