Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
95 lines
5.4 KiB
Markdown
95 lines
5.4 KiB
Markdown
# Phase 6 — Load testing with k6, and percentiles
|
||
|
||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
|
||
|
||
**Status:** Not started
|
||
|
||
**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
|
||
benefits from Phase 5's dashboards existing.
|
||
|
||
**Time-box:** 1–2 evenings.
|
||
|
||
### Concepts — P95 and P99 properly
|
||
|
||
You asked about these specifically, and there's more to them than "the 95th
|
||
percentile."
|
||
|
||
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
|
||
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
|
||
is.
|
||
- **You cannot average percentiles.** Not across instances, not across time buckets.
|
||
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
|
||
to aggregate the *histogram buckets* and compute the quantile from the sum:
|
||
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
|
||
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
|
||
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
|
||
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
|
||
bracket your actual SLO threshold. (Native histograms largely solve this and are
|
||
worth knowing about.)
|
||
- **Which population?** An overall p99 across all endpoints is dominated by whichever
|
||
endpoint has the most traffic. Slice by route, or you'll miss that
|
||
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
|
||
- **Tail latency amplification** — the most important idea here for a microservice
|
||
system. If one user request fans out to 10 backend calls, each with an independent
|
||
p99 of 1 s, then the chance the *user request* touches at least one slow call is
|
||
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
|
||
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
|
||
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
|
||
measurable — go measure it.
|
||
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
|
||
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
|
||
that.
|
||
- **Coordinated omission** — a load-generator trap. If your generator waits for a
|
||
response before sending the next request, then when the system stalls, the
|
||
generator stalls too and simply *doesn't record* the requests that would have been
|
||
slow. Your measured latency is systematically optimistic. Fix: use an **open
|
||
model** — a fixed *arrival rate* independent of response times. In k6 that means
|
||
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
|
||
`constant-vus`. Knowing this is a genuine senior-level differentiator.
|
||
|
||
### Concepts — k6
|
||
|
||
- **Test types:** smoke (does it work at all), average-load (normal traffic),
|
||
stress (past normal, find the knee), spike (sudden surge, test recovery),
|
||
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
|
||
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
|
||
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
|
||
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
|
||
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
|
||
fail the run; **thresholds are SLOs expressed in code and they set the process exit
|
||
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
|
||
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
|
||
view and the application's view sit on the same Grafana time axis. That side-by-side
|
||
is the single most convincing screenshot you can show Babak.
|
||
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
|
||
writing 60 lines of it.
|
||
|
||
### Resources
|
||
|
||
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
|
||
*Thresholds*. Free, genuinely well written; ~2 h to work through.
|
||
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
|
||
hands-on).
|
||
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
|
||
- Grafana docs → *k6 output to Prometheus remote write*.
|
||
|
||
### Build
|
||
|
||
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
|
||
sharing a `common.js` for setup and payload generation against random accounts.
|
||
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
|
||
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
|
||
`http_req_failed rate<0.01`.
|
||
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
|
||
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
|
||
sets. This before/after pair is your baseline for Phase 8.
|
||
6. Optional but impressive: have k6 send W3C trace context headers so load-test
|
||
requests appear as traces, letting you click from a k6 spike into a specific
|
||
trace. Check the current k6 docs for the tracing module's status — it has moved
|
||
between experimental and stable.
|
||
|
||
**Done when:** `make load-stress` produces a Grafana view where you can see the
|
||
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
|
||
service that broke first.
|
||
|