From b50f8c3b2413a56cf9d8acf199966b8634fe1e01 Mon Sep 17 00:00:00 2001 From: Movses Aslanyan Date: Sun, 25 Oct 2026 02:04:54 +0330 Subject: [PATCH] Mark Phase 5 done RED per service, the service map and exemplar click-through are verified live. Build step 3's second metrics path and step 6's TraceQL test migration are not done. Co-Authored-By: Claude Opus 5.5 --- docs/roadmap/phase-5.md | 46 +++++++++++++++++++++++++++++++++++++++-- docs/roadmap/summary.md | 5 +++-- 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/docs/roadmap/phase-5.md b/docs/roadmap/phase-5.md index 0dd2ad3..cd1d214 100644 --- a/docs/roadmap/phase-5.md +++ b/docs/roadmap/phase-5.md @@ -2,9 +2,51 @@ *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)* -**Status:** In progress +**Status:** Done -**Message:** Started 2026-10-13. +**Message:** Done 2026-10-25. All three Done-when criteria verified live: RED per +service and the service map (7 edges, including the two stream edges +gateway → notifier and gateway → audit) from Tempo's span metrics, and a Grafana +exemplar click that opens the trace. + +- **Build steps done:** 1, 2, 3 (first path only), 4 and 5. Not done: 3's second + path and 6 (see below). +- **Metrics path:** each service exposes OTel metrics through + `PrometheusMetricReader` on its own `:9464` (`common/telemetry.py`), not an OTLP + exporter, because Alloy 404s OTLP metrics. Prometheus's `ledgerline` job scrapes + them through czy's Traefik at `https://ledgerline.vione.cloud/metrics/` + behind basic auth. +- **Custom metrics:** the five from Build step 2, in `services/*/app.py` and + `common/metrics.py`. `ledgerline.db.pool.in_use` is labelled `pool` + (`normal`/`starved`/`default`), not `service`; the scrape adds `service`. + `currency` outside EUR/USD/GBP is recorded as `other`. +- **Cardinality experiment (Build step 4, done):** `account_id` on the transaction counter turned 75 + transactions into 68 series across 60 accounts, and `account_id` entered the + TSDB Status top-10. After the revert (`d54ea2b`) the series went stale, but head + series stayed up until compaction. +- **Hand-written vs span-metrics histogram** (same 75 requests): counts equal, + `_sum` 400.85 s vs 401.39 s (the span adds ~7 ms per request of middleware and + serialization). p95 was 9.5 s vs 14.6 s. The gap comes only from Tempo's wide + 8.192–16.384 s bucket. +- **Exemplar fix:** `trace_id_label_name: trace_id` in `k8s/tempo/tempo.yml`. + Tempo's code default is `traceID`; its docs wrongly say `trace_id`. +- **Not done:** Build step 3's second metrics path and its write-up (a Phase 10 + artefact), and Build step 6 (migrating the call-counting tests to TraceQL). +- **Carry forward:** + - RED must count `SPAN_KIND_SERVER|SPAN_KIND_CONSUMER`. Notifier and audit only + have CONSUMER spans. + - An error-ratio query needs `(errors or total * 0) / total`, or services with + zero errors drop out. + - For gateway's latency tail, use `ledgerline_transaction_duration_seconds`. + The span-metrics p95/p99 there is an interpolation artifact. + - The Python SDK has no cardinality limit, and the Python Prometheus exporter + emits no exemplars, so only Tempo's span metrics carry them. + - Built-in HTTP histograms are most of the app's series. The body-size ones + (~⅓) can be dropped with a View or `metric_relabel_configs` if needed. + - Exemplars stored before 2026-10-24 19:10 UTC are labelled `traceID` and don't + link. Exemplar trace IDs drop leading zeros. + - Tempo stops sending idle service-graph and span-metric series after a few + minutes, so "now" queries come back empty without traffic. **Time-box:** 2 evenings. diff --git a/docs/roadmap/summary.md b/docs/roadmap/summary.md index 299b4cb..1a84bf4 100644 --- a/docs/roadmap/summary.md +++ b/docs/roadmap/summary.md @@ -20,7 +20,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. | 2 | [`phase-2.md`](phase-2.md) | Done | Concepts + first trace | 1 evening | One trace in Tempo | | 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | | 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation | -| 5 | [`phase-5.md`](phase-5.md) | In progress | Metrics | 2 evenings | RED per service, service map, exemplars | +| 5 | [`phase-5.md`](phase-5.md) | Done | Metrics | 2 evenings | RED per service, service map, exemplars | | 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards | | 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles | | 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts | @@ -52,7 +52,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. - **5 — Metrics.** Instrument RED metrics per service, understand cardinality limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics (404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the - service graph — don't duplicate it in the collector. + service graph — don't duplicate it in the collector. **Done** — see + `phase-5.md` (second metrics path and the TraceQL test migration not done). - **6 — Dashboards and alerts as code.** Generate dashboard JSON with the Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so "destroy and recreate in seconds" is a real, repeatable command.