Add deployment phase and Traefik compose override
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b2112e55e1
commit
5e514ecf25
+75
-74
@@ -1,90 +1,91 @@
|
||||
# Phase 8 — Bottleneck hunting (the capstone)
|
||||
# Phase 8 — SLI, SLO, SLA, error budgets, RTO, RPO
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
|
||||
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
|
||||
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
|
||||
**Message:** Depends on Phase 7's load-test baseline (the latency SLO should
|
||||
come from measured numbers, not be picked from thin air).
|
||||
|
||||
**Time-box:** 2 evenings. This is where the whole thing pays off.
|
||||
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
|
||||
|
||||
### Concepts — how to actually answer "how do I find bottlenecks?"
|
||||
### Concepts
|
||||
|
||||
Babak asked how you'd *think* about this, not for a list of tools. Here's the
|
||||
method, roughly in order. The discipline is to go **top-down from the user, not
|
||||
bottom-up from the infrastructure** — bottom-up is how people spend three hours
|
||||
tuning a database that wasn't the problem.
|
||||
|
||||
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
|
||||
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
|
||||
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
|
||||
2. **Localise to a service** using RED + the service graph. Which node's duration
|
||||
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
|
||||
the graph from the leaves inward.
|
||||
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
|
||||
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
|
||||
several slow traces plus one fast trace of the same endpoint as a control.
|
||||
4. **Read the trace for five specific shapes:**
|
||||
- **A long span** — the work itself is slow. Look at its attributes: which query,
|
||||
which host, which parameters.
|
||||
- **A gap** — wall-clock time not covered by any span. This is the interesting
|
||||
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
|
||||
blocked event loop, or a missing instrumentation library. Ledgerline's
|
||||
`pool_starvation` and `blocking_event_loop` both show up as gaps.
|
||||
- **Serialisation** — spans laid end to end that could have been concurrent.
|
||||
Ledgerline's `no_batching` looks exactly like this.
|
||||
- **Repetition** — the same span name 150 times in a row. That's your N+1.
|
||||
- **Retries** — the same client span repeated 3–4 times with errors. That's
|
||||
`retry_storm`, and note that it makes an *overload* worse, never better.
|
||||
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
|
||||
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
|
||||
memory. A resource at 100% utilisation with a growing queue is saturated;
|
||||
latency there is a *symptom*, not the cause.
|
||||
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
|
||||
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
|
||||
concurrent slots. If your pool is 5, you queue — and queueing latency grows
|
||||
non-linearly as you approach saturation. This one formula explains most
|
||||
connection-pool and worker-count incidents you'll ever see.
|
||||
7. **Check for tail amplification** (Phase 6) before blaming any single service.
|
||||
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
|
||||
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
|
||||
over the *same duration*. One variable at a time. Never declare a fix from a
|
||||
dashboard that "looks better."
|
||||
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
|
||||
becomes the bottleneck. Re-measure after every change; stop when you're inside
|
||||
the SLO, not when you run out of ideas.
|
||||
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
|
||||
or lock waits inside Postgres, or which line of Python burned the cycles. That's
|
||||
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
|
||||
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
|
||||
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
|
||||
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
|
||||
`POST /api/transactions` requests that returned 2xx within 500 ms.*
|
||||
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
|
||||
requests succeed within 500 ms, measured over a rolling 28 days."
|
||||
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
|
||||
Always **looser** than your SLO, so you have room to miss internally without
|
||||
breaching commercially. If your SLA and SLO are the same number, you've made a
|
||||
mistake.
|
||||
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
|
||||
budget; over 30 days that's about **43 minutes**. The budget is the thing that
|
||||
turns reliability from an argument into arithmetic: if there's budget left, ship
|
||||
features and take risks; if it's spent, the policy says you stop feature work and
|
||||
spend on reliability. That policy — agreed in advance by dev, ops and product — is
|
||||
the whole point.
|
||||
- **Burn rate.** How fast you're consuming budget relative to the window.
|
||||
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
|
||||
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
|
||||
is what stops you from either paging on every blip or noticing the slow bleed a
|
||||
month late.
|
||||
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
|
||||
and it's worth being precise because interviewers probe exactly here.
|
||||
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
|
||||
from failure to restored service. It drives your architecture: replication,
|
||||
warm standby, failover automation, runbook quality.
|
||||
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
|
||||
in time: how far back can you afford to lose. It drives your **backup and
|
||||
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
|
||||
every 15 minutes.
|
||||
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
|
||||
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
|
||||
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
|
||||
5 minutes of transactions." Both feed the SLA.
|
||||
- Both are **business decisions, not technical ones** — you derive them by asking
|
||||
what an hour of downtime or an hour of lost data actually costs, then buying the
|
||||
cheapest architecture that meets it. Tightening either is expensive and
|
||||
non-linear.
|
||||
|
||||
### Resources
|
||||
|
||||
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
|
||||
for saturation.
|
||||
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
|
||||
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
|
||||
thing ever written on why microservice latency behaves the way it does.
|
||||
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
|
||||
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
|
||||
these; connect them to the trace's DB spans.
|
||||
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
|
||||
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
|
||||
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
|
||||
all free PDFs. Run the exercises solo with Ledgerline as the subject.
|
||||
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
|
||||
free, self-paced, deeper.
|
||||
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
|
||||
overview; pair it with your Postgres backup knowledge.
|
||||
|
||||
### Build
|
||||
|
||||
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
|
||||
`docs/PATHOLOGIES.md`.**
|
||||
2. Run `make load-average` for 10 minutes.
|
||||
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
|
||||
like a real incident doc: what you observed, what you hypothesised, what you
|
||||
checked, what it turned out to be. Include screenshots.
|
||||
4. When you think you've found them all, run `make chaos-clean`, enable them one at
|
||||
a time, and confirm each signature.
|
||||
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
|
||||
6. Repeat once with a new random set. The second run is where the method actually
|
||||
becomes yours.
|
||||
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
|
||||
- Availability SLI/SLO on `POST /api/transactions`
|
||||
- Latency SLI/SLO on the same (pick the threshold from your Phase 7 baseline, not
|
||||
from thin air)
|
||||
- A freshness/lag SLI on the notifier consumer group
|
||||
- A stated SLA that is deliberately looser than each SLO
|
||||
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
|
||||
- An error budget policy: what happens at 50%, 75% and 100% consumed
|
||||
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
|
||||
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
|
||||
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
|
||||
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
|
||||
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
|
||||
be able to argue the alternative: Grafana unified alerting can evaluate across
|
||||
datasources (including Loki queries) and lives with your dashboards, whereas
|
||||
Alertmanager is metrics-only but is the ecosystem standard and gives you
|
||||
grouping, deduplication, inhibition and silences for free. Note which you chose
|
||||
and why in `docs/SLO.md`.
|
||||
3. Add an error-budget panel to the overview dashboard: budget remaining, current
|
||||
burn rate, projected exhaustion.
|
||||
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
|
||||
watch the fast-burn alert fire.
|
||||
|
||||
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
|
||||
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
|
||||
**Done when:** you can explain, in one paragraph each and without notes, the
|
||||
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
|
||||
decision.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user