Add deployment phase and Traefik compose override
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b2112e55e1
commit
5e514ecf25
+60
-78
@@ -1,94 +1,76 @@
|
||||
# Phase 6 — Load testing with k6, and percentiles
|
||||
# Phase 6 — Dashboards and alerts as code
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
|
||||
benefits from Phase 5's dashboards existing.
|
||||
**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
|
||||
dashboarding.
|
||||
|
||||
**Time-box:** 1–2 evenings.
|
||||
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
|
||||
recreate in seconds.*
|
||||
|
||||
### Concepts — P95 and P99 properly
|
||||
### Concepts
|
||||
|
||||
You asked about these specifically, and there's more to them than "the 95th
|
||||
percentile."
|
||||
|
||||
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
|
||||
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
|
||||
is.
|
||||
- **You cannot average percentiles.** Not across instances, not across time buckets.
|
||||
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
|
||||
to aggregate the *histogram buckets* and compute the quantile from the sum:
|
||||
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
|
||||
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
|
||||
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
|
||||
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
|
||||
bracket your actual SLO threshold. (Native histograms largely solve this and are
|
||||
worth knowing about.)
|
||||
- **Which population?** An overall p99 across all endpoints is dominated by whichever
|
||||
endpoint has the most traffic. Slice by route, or you'll miss that
|
||||
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
|
||||
- **Tail latency amplification** — the most important idea here for a microservice
|
||||
system. If one user request fans out to 10 backend calls, each with an independent
|
||||
p99 of 1 s, then the chance the *user request* touches at least one slow call is
|
||||
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
|
||||
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
|
||||
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
|
||||
measurable — go measure it.
|
||||
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
|
||||
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
|
||||
that.
|
||||
- **Coordinated omission** — a load-generator trap. If your generator waits for a
|
||||
response before sending the next request, then when the system stalls, the
|
||||
generator stalls too and simply *doesn't record* the requests that would have been
|
||||
slow. Your measured latency is systematically optimistic. Fix: use an **open
|
||||
model** — a fixed *arrival rate* independent of response times. In k6 that means
|
||||
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
|
||||
`constant-vus`. Knowing this is a genuine senior-level differentiator.
|
||||
|
||||
### Concepts — k6
|
||||
|
||||
- **Test types:** smoke (does it work at all), average-load (normal traffic),
|
||||
stress (past normal, find the knee), spike (sudden surge, test recovery),
|
||||
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
|
||||
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
|
||||
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
|
||||
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
|
||||
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
|
||||
fail the run; **thresholds are SLOs expressed in code and they set the process exit
|
||||
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
|
||||
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
|
||||
view and the application's view sit on the same Grafana time axis. That side-by-side
|
||||
is the single most convincing screenshot you can show Babak.
|
||||
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
|
||||
writing 60 lines of it.
|
||||
- Three levels of "as code", and you should know when each applies:
|
||||
1. **File provisioning** — YAML in `provisioning/datasources/` and
|
||||
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
|
||||
container. Grafana loads them at startup. Simplest, no state, no API tokens,
|
||||
perfect for a Compose demo. **This alone satisfies the requirement.**
|
||||
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
|
||||
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
|
||||
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
|
||||
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
|
||||
of Grafonnet (which is legacy and unofficial).
|
||||
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
|
||||
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
|
||||
when this lands on your Kubernetes cluster.
|
||||
- **Dashboards should answer questions, not display data.** A good service dashboard
|
||||
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
|
||||
is the pressure. Panels that nobody looks at during an incident are cost, not value.
|
||||
- **Alerts as code too.** Grafana alert rules, contact points and notification
|
||||
policies all provision from YAML. And keep the *alert definition* next to the
|
||||
dashboard that explains it.
|
||||
|
||||
### Resources
|
||||
|
||||
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
|
||||
*Thresholds*. Free, genuinely well written; ~2 h to work through.
|
||||
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
|
||||
hands-on).
|
||||
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
|
||||
- Grafana docs → *k6 output to Prometheus remote write*.
|
||||
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
|
||||
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
|
||||
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
|
||||
example.
|
||||
- Udemy backbone: dashboard design sections (panel types, transformations,
|
||||
variables, dashboard best practice).
|
||||
|
||||
### Build
|
||||
|
||||
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
|
||||
sharing a `common.js` for setup and payload generation against random accounts.
|
||||
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
|
||||
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
|
||||
`http_req_failed rate<0.01`.
|
||||
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
|
||||
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
|
||||
sets. This before/after pair is your baseline for Phase 8.
|
||||
6. Optional but impressive: have k6 send W3C trace context headers so load-test
|
||||
requests appear as traces, letting you click from a k6 spike into a specific
|
||||
trace. Check the current k6 docs for the tracing module's status — it has moved
|
||||
between experimental and stable.
|
||||
Your platform provisions **datasources only** — there is no dashboards provider at
|
||||
all. So this phase is entirely yours, and the right home for it is inside the
|
||||
Ansible role, using the same mechanism as everything else.
|
||||
|
||||
**Done when:** `make load-stress` produces a Grafana view where you can see the
|
||||
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
|
||||
service that broke first.
|
||||
1. **Extend the role.** Add
|
||||
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
|
||||
(a file-based dashboard provider pointing at a mounted directory), a
|
||||
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
|
||||
and copy them. The Compose template already mounts
|
||||
`./config/grafana/provisioning` read-only, so the provider is picked up on
|
||||
restart.
|
||||
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
|
||||
Foundation SDK (Python)**: one function producing a RED dashboard for a given
|
||||
service name, looped over the seven services; a second producing a system
|
||||
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
|
||||
the generator *and* its output; the Ansible role ships the output.
|
||||
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
|
||||
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
|
||||
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
|
||||
than a local Makefile target, because it's the same command that builds the
|
||||
platform.
|
||||
4. While you're in there, review the provisioned datasources and add what's missing:
|
||||
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
|
||||
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
|
||||
understand why `filterByTraceID: true` alone is sufficient here, and when it
|
||||
wouldn't be.
|
||||
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.
|
||||
|
||||
**Done when:** re-running the playbook restores every dashboard from scratch, and no
|
||||
dashboard has ever been saved through the Grafana UI.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user