Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
5.3 KiB
5.3 KiB
Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-6.md · Next: phase-8.md
Status: Not started
Message: Depends on Phase 6's load-test baseline (the latency SLO should come from measured numbers, not be picked from thin air).
Time-box: 1–2 evenings. Mostly thinking, a little config.
Concepts
- SLI — a measurement of something users experience, expressed as a ratio of
good events to valid events. Not "CPU is at 60%". Rather: the proportion of
POST /api/transactionsrequests that returned 2xx within 500 ms. - SLO — an internal target on an SLI over a window. "99.5% of transaction requests succeed within 500 ms, measured over a rolling 28 days."
- SLA — an external, contractual promise, with consequences (refunds, credits). Always looser than your SLO, so you have room to miss internally without breaching commercially. If your SLA and SLO are the same number, you've made a mistake.
- Error budget —
1 − SLO, expressed as an allowance. A 99.9% SLO gives a 0.1% budget; over 30 days that's about 43 minutes. The budget is the thing that turns reliability from an argument into arithmetic: if there's budget left, ship features and take risks; if it's spent, the policy says you stop feature work and spend on reliability. That policy — agreed in advance by dev, ops and product — is the whole point. - Burn rate. How fast you're consuming budget relative to the window. Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket) is what stops you from either paging on every blip or noticing the slow bleed a month late.
- RTO / RPO — these are disaster recovery metrics, a different axis from SLOs,
and it's worth being precise because interviewers probe exactly here.
- RTO (Recovery Time Objective) — the maximum acceptable downtime: how long from failure to restored service. It drives your architecture: replication, warm standby, failover automation, runbook quality.
- RPO (Recovery Point Objective) — the maximum acceptable data loss, measured in time: how far back can you afford to lose. It drives your backup and replication frequency. RPO of 15 minutes means backups/WAL shipping at least every 15 minutes.
- The relationship worth stating out loud: SLO governs steady-state quality; RTO and RPO govern catastrophe. An SLO says "99.5% of requests are fast." RTO says "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most 5 minutes of transactions." Both feed the SLA.
- Both are business decisions, not technical ones — you derive them by asking what an hour of downtime or an hour of lost data actually costs, then buying the cheapest architecture that meets it. Tightening either is expensive and non-linear.
Resources
- Google SRE Workbook → Implementing SLOs (the SLI menu by service type is immediately usable), then Alerting on SLOs, then Error Budget Policy. Free.
- Google SRE — The Art of SLOs: slides plus participant and facilitator handbooks, all free PDFs. Run the exercises solo with Ledgerline as the subject.
- Optional: Coursera Measuring and Managing Reliability (Google Cloud) — auditable free, self-paced, deeper.
- For RTO/RPO: Splunk's RPO vs RTO explainer is a clean, vendor-neutral-enough overview; pair it with your Postgres backup knowledge.
Build
- Write
docs/SLO.md. For Ledgerline define, with justification:- Availability SLI/SLO on
POST /api/transactions - Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not from thin air)
- A freshness/lag SLI on the notifier consumer group
- A stated SLA that is deliberately looser than each SLO
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
- An error budget policy: what happens at 50%, 75% and 100% consumed
- Availability SLI/SLO on
- Implement recording rules for the SLIs and multi-burn-rate alert rules.
Your platform ships Alertmanager plus a Prometheus rules directory and the
sample rules
HostExporterDownandHostHighCPU, so the grain is Prometheus rule files → Alertmanager, templated by the Ansible role, withalertmanager_webhook_urlingroup_varsfor delivery. Follow the grain — but be able to argue the alternative: Grafana unified alerting can evaluate across datasources (including Loki queries) and lives with your dashboards, whereas Alertmanager is metrics-only but is the ecosystem standard and gives you grouping, deduplication, inhibition and silences for free. Note which you chose and why indocs/SLO.md. - Add an error-budget panel to the overview dashboard: budget remaining, current burn rate, projected exhaustion.
- Prove it: run
chaos-nightmareunder load until you visibly burn budget, and watch the fast-burn alert fire.
Done when: you can explain, in one paragraph each and without notes, the difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule decision.