LedgerLine/docs/roadmap/phase-8.md

# Phase 8 — Bottleneck hunting (the capstone)

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*

**Status:** Not started

**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.

**Time-box:** 2 evenings. This is where the whole thing pays off.

### Concepts — how to actually answer "how do I find bottlenecks?"

Babak asked how you'd *think* about this, not for a list of tools. Here's the
method, roughly in order. The discipline is to go **top-down from the user, not
bottom-up from the infrastructure** — bottom-up is how people spend three hours
tuning a database that wasn't the problem.

1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
   window, compared to what baseline? "It's slow" is not a starting point. "p95 of
   `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
2. **Localise to a service** using RED + the service graph. Which node's duration
   grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
   the graph from the leaves inward.
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
   TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
   several slow traces plus one fast trace of the same endpoint as a control.
4. **Read the trace for five specific shapes:**
   - **A long span** — the work itself is slow. Look at its attributes: which query,
     which host, which parameters.
   - **A gap** — wall-clock time not covered by any span. This is the interesting
     one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
     blocked event loop, or a missing instrumentation library. Ledgerline's
     `pool_starvation` and `blocking_event_loop` both show up as gaps.
   - **Serialisation** — spans laid end to end that could have been concurrent.
     Ledgerline's `no_batching` looks exactly like this.
   - **Repetition** — the same span name 150 times in a row. That's your N+1.
   - **Retries** — the same client span repeated 3–4 times with errors. That's
     `retry_storm`, and note that it makes an *overload* worse, never better.
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
   resource: connection pool in-use vs pool size, stream consumer lag, CPU,
   memory. A resource at 100% utilisation with a growing queue is saturated;
   latency there is a *symptom*, not the cause.
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
   equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
   concurrent slots. If your pool is 5, you queue — and queueing latency grows
   non-linearly as you approach saturation. This one formula explains most
   connection-pool and worker-count incidents you'll ever see.
7. **Check for tail amplification** (Phase 6) before blaming any single service.
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
   chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
   over the *same duration*. One variable at a time. Never declare a fix from a
   dashboard that "looks better."
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
   becomes the bottleneck. Re-measure after every change; stop when you're inside
   the SLO, not when you run out of ideas.
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
    or lock waits inside Postgres, or which line of Python burned the cycles. That's
    where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
    profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.

### Resources

- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
  for saturation.
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
  thing ever written on why microservice latency behaves the way it does.
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
  these; connect them to the trace's DB spans.

### Build

1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
   `docs/PATHOLOGIES.md`.**
2. Run `make load-average` for 10 minutes.
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
   like a real incident doc: what you observed, what you hypothesised, what you
   checked, what it turned out to be. Include screenshots.
4. When you think you've found them all, run `make chaos-clean`, enable them one at
   a time, and confirm each signature.
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
6. Repeat once with a new random set. The second run is where the method actually
   becomes yours.

**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.