Add project documentation and Claude instructions
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 6's load-test baseline (the latency SLO should
|
||||
come from measured numbers, not be picked from thin air).
|
||||
|
||||
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
|
||||
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
|
||||
`POST /api/transactions` requests that returned 2xx within 500 ms.*
|
||||
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
|
||||
requests succeed within 500 ms, measured over a rolling 28 days."
|
||||
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
|
||||
Always **looser** than your SLO, so you have room to miss internally without
|
||||
breaching commercially. If your SLA and SLO are the same number, you've made a
|
||||
mistake.
|
||||
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
|
||||
budget; over 30 days that's about **43 minutes**. The budget is the thing that
|
||||
turns reliability from an argument into arithmetic: if there's budget left, ship
|
||||
features and take risks; if it's spent, the policy says you stop feature work and
|
||||
spend on reliability. That policy — agreed in advance by dev, ops and product — is
|
||||
the whole point.
|
||||
- **Burn rate.** How fast you're consuming budget relative to the window.
|
||||
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
|
||||
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
|
||||
is what stops you from either paging on every blip or noticing the slow bleed a
|
||||
month late.
|
||||
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
|
||||
and it's worth being precise because interviewers probe exactly here.
|
||||
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
|
||||
from failure to restored service. It drives your architecture: replication,
|
||||
warm standby, failover automation, runbook quality.
|
||||
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
|
||||
in time: how far back can you afford to lose. It drives your **backup and
|
||||
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
|
||||
every 15 minutes.
|
||||
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
|
||||
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
|
||||
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
|
||||
5 minutes of transactions." Both feed the SLA.
|
||||
- Both are **business decisions, not technical ones** — you derive them by asking
|
||||
what an hour of downtime or an hour of lost data actually costs, then buying the
|
||||
cheapest architecture that meets it. Tightening either is expensive and
|
||||
non-linear.
|
||||
|
||||
### Resources
|
||||
|
||||
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
|
||||
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
|
||||
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
|
||||
all free PDFs. Run the exercises solo with Ledgerline as the subject.
|
||||
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
|
||||
free, self-paced, deeper.
|
||||
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
|
||||
overview; pair it with your Postgres backup knowledge.
|
||||
|
||||
### Build
|
||||
|
||||
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
|
||||
- Availability SLI/SLO on `POST /api/transactions`
|
||||
- Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
|
||||
from thin air)
|
||||
- A freshness/lag SLI on the notifier consumer group
|
||||
- A stated SLA that is deliberately looser than each SLO
|
||||
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
|
||||
- An error budget policy: what happens at 50%, 75% and 100% consumed
|
||||
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
|
||||
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
|
||||
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
|
||||
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
|
||||
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
|
||||
be able to argue the alternative: Grafana unified alerting can evaluate across
|
||||
datasources (including Loki queries) and lives with your dashboards, whereas
|
||||
Alertmanager is metrics-only but is the ecosystem standard and gives you
|
||||
grouping, deduplication, inhibition and silences for free. Note which you chose
|
||||
and why in `docs/SLO.md`.
|
||||
3. Add an error-budget panel to the overview dashboard: budget remaining, current
|
||||
burn rate, projected exhaustion.
|
||||
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
|
||||
watch the fast-burn alert fire.
|
||||
|
||||
**Done when:** you can explain, in one paragraph each and without notes, the
|
||||
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
|
||||
decision.
|
||||
|
||||
Reference in New Issue
Block a user