LedgerLine/docs/roadmap/phase-9.md

Phase 9 — Bottleneck hunting (the capstone)

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-8.md · Next: phase-10.md

Status: Not started

Message: Depends on Phases 3–8 all being in place (traces, logs, metrics, dashboards, load-test baselines, and burn-rate alerts are the toolkit this phase exercises). Spoiler rules in CLAUDE.md apply once this phase starts.

Time-box: 2 evenings. This is where the whole thing pays off.

Concepts — how to actually answer "how do I find bottlenecks?"

Babak asked how you'd think about this, not for a list of tools. Here's the method, roughly in order. The discipline is to go top-down from the user, not bottom-up from the infrastructure — bottom-up is how people spend three hours tuning a database that wasn't the problem.

1. Start at the SLI, not the server. Which endpoint, which percentile, which window, compared to what baseline? "It's slow" is not a starting point. "p95 of POST /api/transactions went from 180 ms to 2.1 s at 14:05" is.
2. Localise to a service using RED + the service graph. Which node's duration grew? Note that a slow downstream makes its upstream look slow too, so read the graph from the leaves inward.
3. Get to exemplar traces. Click the latency spike through to a trace, or query TraceQL directly ({ duration > 2s }, { span.status = error }). You want several slow traces plus one fast trace of the same endpoint as a control.
4. Read the trace for five specific shapes:
   • A long span — the work itself is slow. Look at its attributes: which query, which host, which parameters.
   • A gap — wall-clock time not covered by any span. This is the interesting one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a blocked event loop, or a missing instrumentation library. Ledgerline's pool_starvation and blocking_event_loop both show up as gaps.
   • Serialisation — spans laid end to end that could have been concurrent. Ledgerline's no_batching looks exactly like this.
   • Repetition — the same span name 150 times in a row. That's your N+1.
   • Retries — the same client span repeated 3–4 times with errors. That's retry_storm, and note that it makes an overload worse, never better.
5. Separate latency from saturation. Switch from RED to USE on the suspected resource: connection pool in-use vs pool size, stream consumer lag, CPU, memory. A resource at 100% utilisation with a growing queue is saturated; latency there is a symptom, not the cause.
6. Use Little's Law to reason about capacity. L = λ × W: required concurrency equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10 concurrent slots. If your pool is 5, you queue — and queueing latency grows non-linearly as you approach saturation. This one formula explains most connection-pool and worker-count incidents you'll ever see.
7. Check for tail amplification (Phase 7) before blaming any single service.
8. Form one hypothesis and test it with a controlled change. Toggle exactly one chaos flag, re-run the identical k6 profile, and compare the same percentile over the same duration. One variable at a time. Never declare a fix from a dashboard that "looks better."
9. Expect the bottleneck to move. Fix the slowest thing and the second-slowest becomes the bottleneck. Re-measure after every change; stop when you're inside the SLO, not when you run out of ideas.
10. Know what your tools can't see. Traces won't show you CPU time inside a span, or lock waits inside Postgres, or which line of Python burned the cycles. That's where DB-side tools (pg_stat_statements, EXPLAIN ANALYZE) and continuous profiling (Pyroscope, bundled in grafana/otel-lgtm) take over.

Resources

• Brendan Gregg — The USE Method (free article). Short, and the canonical framing for saturation.
• Tom Wilkie / Grafana — The RED Method (free article + conference talk).
• The Tail at Scale (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best thing ever written on why microservice latency behaves the way it does.
• Grafana docs → Tempo → TraceQL. You'll live in this.
• Postgres: EXPLAIN (ANALYZE, BUFFERS) and pg_stat_statements — you already know these; connect them to the trace's DB spans.

Build

1. make chaos-random (a random 3–6 flags, seed never logged). Do not look at docs/PATHOLOGIES.md.
2. Run make load-average for 10 minutes.
3. Hunt. Write docs/INVESTIGATION.md as you go — timestamped, hypothesis-driven, like a real incident doc: what you observed, what you hypothesised, what you checked, what it turned out to be. Include screenshots.
4. When you think you've found them all, run make chaos-clean, enable them one at a time, and confirm each signature.
5. Then read docs/PATHOLOGIES.md and score yourself.
6. Repeat once with a new random set. The second run is where the method actually becomes yours.

Done when: you find at least 4 of the enabled pathologies from telemetry alone, and docs/INVESTIGATION.md reads like something you'd hand to a colleague.