LedgerLine/docs/roadmap/phase-7.md

# Phase 7 — Load testing with k6, and percentiles

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*

**Status:** Not started

**Message:** Depends on Phase 5 (metrics to plot k6's output against) and
benefits from Phase 6's dashboards existing.

**Time-box:** 1–2 evenings.

### Concepts — P95 and P99 properly

You asked about these specifically, and there's more to them than "the 95th
percentile."

- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
  is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
  is.
- **You cannot average percentiles.** Not across instances, not across time buckets.
  `avg(p99)` is wrong and produces a number that looks plausible. The correct move is
  to aggregate the *histogram buckets* and compute the quantile from the sum:
  `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
  your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
  bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
  bracket your actual SLO threshold. (Native histograms largely solve this and are
  worth knowing about.)
- **Which population?** An overall p99 across all endpoints is dominated by whichever
  endpoint has the most traffic. Slice by route, or you'll miss that
  `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
- **Tail latency amplification** — the most important idea here for a microservice
  system. If one user request fans out to 10 backend calls, each with an independent
  p99 of 1 s, then the chance the *user request* touches at least one slow call is
  `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
  p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
  fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
  measurable — go measure it.
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
  most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
  that.
- **Coordinated omission** — a load-generator trap. If your generator waits for a
  response before sending the next request, then when the system stalls, the
  generator stalls too and simply *doesn't record* the requests that would have been
  slow. Your measured latency is systematically optimistic. Fix: use an **open
  model** — a fixed *arrival rate* independent of response times. In k6 that means
  the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
  `constant-vus`. Knowing this is a genuine senior-level differentiator.

### Concepts — k6

- **Test types:** smoke (does it work at all), average-load (normal traffic),
  stress (past normal, find the knee), spike (sudden surge, test recovery),
  soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
  Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
  (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
  fail the run; **thresholds are SLOs expressed in code and they set the process exit
  code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
  view and the application's view sit on the same Grafana time axis. That side-by-side
  is the single most convincing screenshot you can show Babak.
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
  writing 60 lines of it.

### Resources

- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
  *Thresholds*. Free, genuinely well written; ~2 h to work through.
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
  hands-on).
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
- Grafana docs → *k6 output to Prometheus remote write*.

### Build

1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
   sharing a `common.js` for setup and payload generation against random accounts.
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
   `http_req_failed rate<0.01`.
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
   sets. This before/after pair is your baseline for Phase 9.
6. Optional but impressive: have k6 send W3C trace context headers so load-test
   requests appear as traces, letting you click from a k6 spike into a specific
   trace. Check the current k6 docs for the tracing module's status — it has moved
   between experimental and stable.

**Done when:** `make load-stress` produces a Grafana view where you can see the
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
service that broke first.