LedgerLine/docs/roadmap/phase-6.md

Phase 6 — Load testing with k6, and percentiles

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-5.md · Next: phase-7.md

Status: Not started

Message: Depends on Phase 4 (metrics to plot k6's output against) and benefits from Phase 5's dashboards existing.

Time-box: 1–2 evenings.

Concepts — P95 and P99 properly

You asked about these specifically, and there's more to them than "the 95th percentile."

• A percentile is meaningless without a population and a window. "p95 = 400 ms" is not a fact; "p95 of POST /api/transactions over the last 5 minutes = 400 ms" is.
• You cannot average percentiles. Not across instances, not across time buckets. avg(p99) is wrong and produces a number that looks plausible. The correct move is to aggregate the histogram buckets and compute the quantile from the sum: histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le)).
• Prometheus classic histograms give you an interpolated estimate, bounded by your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that bracket your actual SLO threshold. (Native histograms largely solve this and are worth knowing about.)
• Which population? An overall p99 across all endpoints is dominated by whichever endpoint has the most traffic. Slice by route, or you'll miss that /api/accounts/{id}/insights is catastrophic while the aggregate looks fine.
• Tail latency amplification — the most important idea here for a microservice system. If one user request fans out to 10 backend calls, each with an independent p99 of 1 s, then the chance the user request touches at least one slow call is 1 − 0.99¹⁰ ≈ 9.6%. Your user-facing p99 is worse than any individual service's p99. This is Dean & Barroso's The Tail at Scale, and it's why "each service is fine" and "the product is slow" coexist. Ledgerline's fan-out makes this measurable — go measure it.
• p99 vs p99.9: the users in the deep tail are often your heaviest, most valuable, most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly that.
• Coordinated omission — a load-generator trap. If your generator waits for a response before sending the next request, then when the system stalls, the generator stalls too and simply doesn't record the requests that would have been slow. Your measured latency is systematically optimistic. Fix: use an open model — a fixed arrival rate independent of response times. In k6 that means the constant-arrival-rate / ramping-arrival-rate executors rather than constant-vus. Knowing this is a genuine senior-level differentiator.

Concepts — k6

• Test types: smoke (does it work at all), average-load (normal traffic), stress (past normal, find the knee), spike (sudden surge, test recovery), soak (hours, find leaks and lag drift), breakpoint (ramp until failure). Your unbounded_cache and slow_consumer pathologies are only visible in a soak.
• Executors: closed model (constant-vus, ramping-vus) vs open model (constant-arrival-rate, ramping-arrival-rate). See coordinated omission above.
• checks vs thresholds: checks are assertions that record pass/fail but don't fail the run; thresholds are SLOs expressed in code and they set the process exit code, which is what makes k6 a CI gate. http_req_duration: ['p(95)<500'].
• Output: k6 can remote-write its own metrics to Prometheus, so the load test's view and the application's view sit on the same Grafana time axis. That side-by-side is the single most convincing screenshot you can show Babak.
• k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're writing 60 lines of it.

Resources

• Grafana k6 docs → Get started, Test types, Scenarios & executors, Thresholds. Free, genuinely well written; ~2 h to work through.
• Grafana workshop — Introduction to Performance Testing with Grafana k6 (free, hands-on).
• Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
• Grafana docs → k6 output to Prometheus remote write.

Build

1. loadtest/ with four scripts: smoke.js, average.js, stress.js, soak.js, sharing a common.js for setup and payload generation against random accounts.
2. Use arrival-rate executors for average and stress. Write down why.
3. Thresholds encoding your intended SLOs: p(95)<500, p(99)<2000, http_req_failed rate<0.01.
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
5. Run the same profile under chaos-clean and chaos-nightmare. Save both result sets. This before/after pair is your baseline for Phase 8.
6. Optional but impressive: have k6 send W3C trace context headers so load-test requests appear as traces, letting you click from a k6 spike into a specific trace. Check the current k6 docs for the tracing module's status — it has moved between experimental and stable.

Done when: make load-stress produces a Grafana view where you can see the arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the service that broke first.