LedgerLine/docs/roadmap/phase-8.md

# Phase 8 — SLI, SLO, SLA, error budgets, RTO, RPO

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*

**Status:** Not started

**Message:** Depends on Phase 7's load-test baseline (the latency SLO should
come from measured numbers, not be picked from thin air).

**Time-box:** 1–2 evenings. Mostly thinking, a little config.

### Concepts

- **SLI** — a *measurement* of something users experience, expressed as a ratio of
  good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
  `POST /api/transactions` requests that returned 2xx within 500 ms.*
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
  requests succeed within 500 ms, measured over a rolling 28 days."
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
  Always **looser** than your SLO, so you have room to miss internally without
  breaching commercially. If your SLA and SLO are the same number, you've made a
  mistake.
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
  budget; over 30 days that's about **43 minutes**. The budget is the thing that
  turns reliability from an argument into arithmetic: if there's budget left, ship
  features and take risks; if it's spent, the policy says you stop feature work and
  spend on reliability. That policy — agreed in advance by dev, ops and product — is
  the whole point.
- **Burn rate.** How fast you're consuming budget relative to the window.
  Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
  multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
  is what stops you from either paging on every blip or noticing the slow bleed a
  month late.
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
  and it's worth being precise because interviewers probe exactly here.
  - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
    from failure to restored service. It drives your architecture: replication,
    warm standby, failover automation, runbook quality.
  - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
    in time: how far back can you afford to lose. It drives your **backup and
    replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
    every 15 minutes.
  - The relationship worth stating out loud: **SLO governs steady-state quality; RTO
    and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
    "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
    5 minutes of transactions." Both feed the SLA.
  - Both are **business decisions, not technical ones** — you derive them by asking
    what an hour of downtime or an hour of lost data actually costs, then buying the
    cheapest architecture that meets it. Tightening either is expensive and
    non-linear.

### Resources

- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
  immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
  all free PDFs. Run the exercises solo with Ledgerline as the subject.
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
  free, self-paced, deeper.
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
  overview; pair it with your Postgres backup knowledge.

### Build

1. Write `docs/SLO.md`. For Ledgerline define, with justification:
   - Availability SLI/SLO on `POST /api/transactions`
   - Latency SLI/SLO on the same (pick the threshold from your Phase 7 baseline, not
     from thin air)
   - A freshness/lag SLI on the notifier consumer group
   - A stated SLA that is deliberately looser than each SLO
   - An RTO and RPO for the Postgres instance, with the backup cadence they imply
   - An error budget policy: what happens at 50%, 75% and 100% consumed
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
   Your platform ships **Alertmanager** plus a Prometheus rules directory and the
   sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
   *Prometheus rule files → Alertmanager*, templated by the Ansible role, with
   `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
   be able to argue the alternative: Grafana unified alerting can evaluate across
   datasources (including Loki queries) and lives with your dashboards, whereas
   Alertmanager is metrics-only but is the ecosystem standard and gives you
   grouping, deduplication, inhibition and silences for free. Note which you chose
   and why in `docs/SLO.md`.
3. Add an error-budget panel to the overview dashboard: budget remaining, current
   burn rate, projected exhaustion.
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
   watch the fast-burn alert fire.

**Done when:** you can explain, in one paragraph each and without notes, the
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
decision.