LedgerLine/docs/roadmap/phase-5.md

# Phase 5 — Metrics

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*

**Status:** Done

**Message:** Done 2026-10-25. All three Done-when criteria verified live: RED per
service and the service map (7 edges, including the two stream edges
gateway → notifier and gateway → audit) from Tempo's span metrics, and a Grafana
exemplar click that opens the trace.

- **Build steps done:** 1, 2, 3 (first path only), 4 and 5. Not done: 3's second
  path and 6 (see below).
- **Metrics path:** each service exposes OTel metrics through
  `PrometheusMetricReader` on its own `:9464` (`common/telemetry.py`), not an OTLP
  exporter, because Alloy 404s OTLP metrics. Prometheus's `ledgerline` job scrapes
  them through czy's Traefik at `https://ledgerline.vione.cloud/metrics/<service>`
  behind basic auth.
- **Custom metrics:** the five from Build step 2, in `services/*/app.py` and
  `common/metrics.py`. `ledgerline.db.pool.in_use` is labelled `pool`
  (`normal`/`starved`/`default`), not `service`; the scrape adds `service`.
  `currency` outside EUR/USD/GBP is recorded as `other`.
- **Cardinality experiment (Build step 4, done):** `account_id` on the transaction counter turned 75
  transactions into 68 series across 60 accounts, and `account_id` entered the
  TSDB Status top-10. After the revert (`d54ea2b`) the series went stale, but head
  series stayed up until compaction.
- **Hand-written vs span-metrics histogram** (same 75 requests): counts equal,
  `_sum` 400.85 s vs 401.39 s (the span adds ~7 ms per request of middleware and
  serialization). p95 was 9.5 s vs 14.6 s. The gap comes only from Tempo's wide
  8.192–16.384 s bucket.
- **Exemplar fix:** `trace_id_label_name: trace_id` in `k8s/tempo/tempo.yml`.
  Tempo's code default is `traceID`; its docs wrongly say `trace_id`.
- **Not done:** Build step 3's second metrics path and its write-up (a Phase 10
  artefact), and Build step 6 (migrating the call-counting tests to TraceQL).
- **Carry forward:**
  - RED must count `SPAN_KIND_SERVER|SPAN_KIND_CONSUMER`. Notifier and audit only
    have CONSUMER spans.
  - An error-ratio query needs `(errors or total * 0) / total`, or services with
    zero errors drop out.
  - For gateway's latency tail, use `ledgerline_transaction_duration_seconds`.
    The span-metrics p95/p99 there is an interpolation artifact.
  - The Python SDK has no cardinality limit, and the Python Prometheus exporter
    emits no exemplars, so only Tempo's span metrics carry them.
  - Built-in HTTP histograms are most of the app's series. The body-size ones
    (~⅓) can be dropped with a View or `metric_relabel_configs` if needed.
  - Exemplars stored before 2026-10-24 19:10 UTC are labelled `traceID` and don't
    link. Exemplar trace IDs drop leading zeros.
  - Tempo stops sending idle service-graph and span-metric series after a few
    minutes, so "now" queries come back empty without traffic.

**Time-box:** 2 evenings.

### Concepts

- **Instrument types.**
  - **Counter** — monotonic, only increases. Requests, errors, messages consumed.
  - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
    connection-pool checkouts.
  - **Histogram** — a distribution. Request latency, payload size. This is what
    gives you percentiles.
  - Synchronous instruments are recorded inline; **observable/asynchronous** ones are
    read via a callback on collection (right for things you *poll*, like pool size).
  - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
    **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
    nonsense. Use histograms.
- **Cardinality — the one thing that will bite you.** Every unique combination of
  metric name + label values is a separate time series in memory. Add `account_id`
  as a label on a 800-account demo and you've made 800 series; do it in production
  and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
  ~10 000 distinct values in production, it does not belong on a metric.** Put it on
  a span. Histograms multiply this (one series per bucket per label combination), so
  don't over-bucket either.
  - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
    the concrete path), `http.status_code` class, `kind`, `currency`.
  - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
    raw URL, timestamp, error message.
- **RED and USE.**
  - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
    per-service dashboard.
  - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
    connection pools, queues, disk. Saturation is the one everyone forgets and the
    one that predicts collapse.
  - You want both: RED tells you the service is slow, USE tells you which resource
    is why.
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
  and knowing both is the point:
  - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
    `servicegraph` connectors read spans in flight and emit RED metrics and edge
    metrics. This is what most blog posts (including the OneUptime article) show.
  - *In the trace store:* Tempo's **metrics-generator** does the same job after
    ingestion, remote-writing the result to Prometheus. **This is what your platform
    does** — `processors: [service-graphs, span-metrics, local-blocks]` with
    `send_exemplars: true`, remote-writing to a Prometheus started with
    `--web.enable-remote-write-receiver`.
  - So RED metrics, the service map and exemplars **already exist** before you write
    a single metric line. Do not add the collector connectors as well — you'd
    duplicate every series under a second set of names. **Go find the real metric
    names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
    they will not match the article's.
  - The trade-off worth being able to argue: collector-side generation is cheaper
    and works before storage, but sees only what passes through that collector;
    Tempo-side generation sees everything that landed, survives collector restarts,
    and can be enabled per-tenant — at the cost of load on the trace store.
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
  click a spike on a latency graph and land in the exact trace. On your platform
  this is wired: Tempo's generator sets `send_exemplars: true`, Prometheus stores
  them (`--enable-feature=exemplar-storage`, added in `k8s/`; the Ansible role
  lacked it), and the Prometheus datasource declares `exemplarTraceIdDestinations`
  with `name: trace_id`. If you emit your own histograms with exemplars, the label
  must be `trace_id` to match. **But not quite end to end:** Tempo's generator
  labels its exemplars `traceID` (verified live), so the span-metric exemplars
  don't match that destination. Find the fix yourself and choose one: the
  datasource, Tempo, or both.
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
  can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
  for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
  the pull model is why Prometheus has service discovery and why `up` exists.

### Resources

- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
  Read the first half. Best cardinality explanation available.
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
  not a fact.
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
  `servicegraph` connector config and its RED PromQL queries almost verbatim.
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.

### Build

1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
2. Custom application metrics, deliberately few:
   - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
   - `ledgerline.transaction.duration` (Histogram; labels: `kind`)
   - `ledgerline.fraud.score` (Histogram; no labels)
   - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
   - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy rejects OTLP
   metrics (404 on `/v1/metrics`), so pick a path and implement it:
   - *Platform grain:* expose `/metrics` on each service and add scrape jobs to
     `k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
     then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
     **First decide how Prometheus on falken reaches czy**: the Traefik
     override strips every service's host port (see `phase-1.md`). Note the irony and the
     lesson — you're back to a pull model and a second client library, which is
     exactly the fragmentation OTel exists to fix.
   - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
     Alloy so app metrics flow through the collector as the architecture drawing
     intends. This also closes a real diagram-vs-code gap in the repo.
   - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
     `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
     all three signals.
   Do the first to get moving; do one of the others afterwards and write up which
   you'd choose for a real platform team and why. That write-up is a Phase 10
   artefact.
4. Deliberately break it: add `account_id` as a label to
   `ledgerline.transactions.total`, run a 200-request burst, and watch the series
   count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
   you'll never forget.
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
   one. Understand why they differ slightly and when you'd want each.
6. **Second migration of the call-counting assertions.** The three tests you moved
   to LogQL in Phase 4 can now count **client spans** in Tempo instead — a
   TraceQL query for `sanctions` client spans within a single trace answers
   "how many lookups did this one transaction make?" directly, which is the
   question the test was always really asking. Do it and compare all three
   generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
   clearest possible demonstration of what each signal is actually for, on a
   problem you already understand.

**Done when:** you have a RED view per service, a working service map, and you can
click a latency spike through to a trace via an exemplar.