Mark Phase 5 done

RED per service, the service map and exemplar click-through are verified live. Build step 3's second metrics path and step 6's TraceQL test migration are not done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 02:04:54 +03:30
parent c2bd962597
commit b50f8c3b24
2 changed files with 47 additions and 4 deletions
@@ -2,9 +2,51 @@

 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*

-**Status:** In progress
+**Status:** Done

-**Message:** Started 2026-10-13.
+**Message:** Done 2026-10-25. All three Done-when criteria verified live: RED per
+service and the service map (7 edges, including the two stream edges
+gateway → notifier and gateway → audit) from Tempo's span metrics, and a Grafana
+exemplar click that opens the trace.
+
+- **Build steps done:** 1, 2, 3 (first path only), 4 and 5. Not done: 3's second
+  path and 6 (see below).
+- **Metrics path:** each service exposes OTel metrics through
+  `PrometheusMetricReader` on its own `:9464` (`common/telemetry.py`), not an OTLP
+  exporter, because Alloy 404s OTLP metrics. Prometheus's `ledgerline` job scrapes
+  them through czy's Traefik at `https://ledgerline.vione.cloud/metrics/<service>`
+  behind basic auth.
+- **Custom metrics:** the five from Build step 2, in `services/*/app.py` and
+  `common/metrics.py`. `ledgerline.db.pool.in_use` is labelled `pool`
+  (`normal`/`starved`/`default`), not `service`; the scrape adds `service`.
+  `currency` outside EUR/USD/GBP is recorded as `other`.
+- **Cardinality experiment (Build step 4, done):** `account_id` on the transaction counter turned 75
+  transactions into 68 series across 60 accounts, and `account_id` entered the
+  TSDB Status top-10. After the revert (`d54ea2b`) the series went stale, but head
+  series stayed up until compaction.
+- **Hand-written vs span-metrics histogram** (same 75 requests): counts equal,
+  `_sum` 400.85 s vs 401.39 s (the span adds ~7 ms per request of middleware and
+  serialization). p95 was 9.5 s vs 14.6 s. The gap comes only from Tempo's wide
+  8.192–16.384 s bucket.
+- **Exemplar fix:** `trace_id_label_name: trace_id` in `k8s/tempo/tempo.yml`.
+  Tempo's code default is `traceID`; its docs wrongly say `trace_id`.
+- **Not done:** Build step 3's second metrics path and its write-up (a Phase 10
+  artefact), and Build step 6 (migrating the call-counting tests to TraceQL).
+- **Carry forward:**
+  - RED must count `SPAN_KIND_SERVER|SPAN_KIND_CONSUMER`. Notifier and audit only
+    have CONSUMER spans.
+  - An error-ratio query needs `(errors or total * 0) / total`, or services with
+    zero errors drop out.
+  - For gateway's latency tail, use `ledgerline_transaction_duration_seconds`.
+    The span-metrics p95/p99 there is an interpolation artifact.
+  - The Python SDK has no cardinality limit, and the Python Prometheus exporter
+    emits no exemplars, so only Tempo's span metrics carry them.
+  - Built-in HTTP histograms are most of the app's series. The body-size ones
+    (~⅓) can be dropped with a View or `metric_relabel_configs` if needed.
+  - Exemplars stored before 2026-10-24 19:10 UTC are labelled `traceID` and don't
+    link. Exemplar trace IDs drop leading zeros.
+  - Tempo stops sending idle service-graph and span-metric series after a few
+    minutes, so "now" queries come back empty without traffic.

 **Time-box:** 2 evenings.