LedgerLine/docs/roadmap/phase-5.md

Phase 5 — Metrics

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-4.md · Next: phase-6.md

Status: Done

Message: Done 2026-10-25. All three Done-when criteria verified live: RED per service and the service map (7 edges, including the two stream edges gateway → notifier and gateway → audit) from Tempo's span metrics, and a Grafana exemplar click that opens the trace.

• Build steps done: 1, 2, 3 (first path only), 4 and 5. Not done: 3's second path and 6 (see below).
• Metrics path: each service exposes OTel metrics through PrometheusMetricReader on its own :9464 (common/telemetry.py), not an OTLP exporter, because Alloy 404s OTLP metrics. Prometheus's ledgerline job scrapes them through czy's Traefik at https://ledgerline.vione.cloud/metrics/<service> behind basic auth.
• Custom metrics: the five from Build step 2, in services/*/app.py and common/metrics.py. ledgerline.db.pool.in_use is labelled pool (normal/starved/default), not service; the scrape adds service. currency outside EUR/USD/GBP is recorded as other.
• Cardinality experiment (Build step 4, done): account_id on the transaction counter turned 75 transactions into 68 series across 60 accounts, and account_id entered the TSDB Status top-10. After the revert (d54ea2b) the series went stale, but head series stayed up until compaction.
• Hand-written vs span-metrics histogram (same 75 requests): counts equal, _sum 400.85 s vs 401.39 s (the span adds ~7 ms per request of middleware and serialization). p95 was 9.5 s vs 14.6 s. The gap comes only from Tempo's wide 8.192–16.384 s bucket.
• Exemplar fix: trace_id_label_name: trace_id in k8s/tempo/tempo.yml. Tempo's code default is traceID; its docs wrongly say trace_id.
• Not done: Build step 3's second metrics path and its write-up (a Phase 10 artefact), and Build step 6 (migrating the call-counting tests to TraceQL).
• Carry forward:
  • RED must count SPAN_KIND_SERVER|SPAN_KIND_CONSUMER. Notifier and audit only have CONSUMER spans.
  • An error-ratio query needs (errors or total * 0) / total, or services with zero errors drop out.
  • For gateway's latency tail, use ledgerline_transaction_duration_seconds. The span-metrics p95/p99 there is an interpolation artifact.
  • The Python SDK has no cardinality limit, and the Python Prometheus exporter emits no exemplars, so only Tempo's span metrics carry them.
  • Built-in HTTP histograms are most of the app's series. The body-size ones (~⅓) can be dropped with a View or metric_relabel_configs if needed.
  • Exemplars stored before 2026-10-24 19:10 UTC are labelled traceID and don't link. Exemplar trace IDs drop leading zeros.
  • Tempo stops sending idle service-graph and span-metric series after a few minutes, so "now" queries come back empty without traffic.

Time-box: 2 evenings.

Concepts

• Instrument types.
  • Counter — monotonic, only increases. Requests, errors, messages consumed.
  • UpDownCounter / Gauge — can go both ways. Queue depth, in-flight requests, connection-pool checkouts.
  • Histogram — a distribution. Request latency, payload size. This is what gives you percentiles.
  • Synchronous instruments are recorded inline; observable/asynchronous ones are read via a callback on collection (right for things you poll, like pool size).
  • Avoid Prometheus Summaries: they compute quantiles client-side and therefore cannot be aggregated across instances. avg(p99_from_summary) is arithmetic nonsense. Use histograms.
• Cardinality — the one thing that will bite you. Every unique combination of metric name + label values is a separate time series in memory. Add account_id as a label on a 800-account demo and you've made 800 series; do it in production and you make millions and OOM Prometheus. Rule of thumb: if a label could exceed ~10 000 distinct values in production, it does not belong on a metric. Put it on a span. Histograms multiply this (one series per bucket per label combination), so don't over-bucket either.
  • Bounded and fine: service, http.route (the template, /accounts/{id}, not the concrete path), http.status_code class, kind, currency.
  • Unbounded and forbidden: account_id, transaction_id, merchant_name, raw URL, timestamp, error message.
• RED and USE.
  • RED (Rate, Errors, Duration) — for request-driven services. This is your per-service dashboard.
  • USE (Utilization, Saturation, Errors) — for resources: CPU, memory, connection pools, queues, disk. Saturation is the one everyone forgets and the one that predicts collapse.
  • You want both: RED tells you the service is slow, USE tells you which resource is why.
• Span metrics — generate metrics from traces. Two architectures exist for this, and knowing both is the point:
  • In the collector: the OpenTelemetry Collector's spanmetrics and servicegraph connectors read spans in flight and emit RED metrics and edge metrics. This is what most blog posts (including the OneUptime article) show.
  • In the trace store: Tempo's metrics-generator does the same job after ingestion, remote-writing the result to Prometheus. This is what your platform does — processors: [service-graphs, span-metrics, local-blocks] with send_exemplars: true, remote-writing to a Prometheus started with --web.enable-remote-write-receiver.
  • So RED metrics, the service map and exemplars already exist before you write a single metric line. Do not add the collector connectors as well — you'd duplicate every series under a second set of names. Go find the real metric names in Prometheus (/api/v1/label/__name__/values) before writing PromQL; they will not match the article's.
  • The trade-off worth being able to argue: collector-side generation is cheaper and works before storage, but sees only what passes through that collector; Tempo-side generation sees everything that landed, survives collector restarts, and can be enabled per-tenant — at the cost of load on the trace store.
• Exemplars. A histogram bucket can carry a sample trace ID. That's what lets you click a spike on a latency graph and land in the exact trace. On your platform this is wired: Tempo's generator sets send_exemplars: true, Prometheus stores them (--enable-feature=exemplar-storage, added in k8s/; the Ansible role lacked it), and the Prometheus datasource declares exemplarTraceIdDestinations with name: trace_id. If you emit your own histograms with exemplars, the label must be trace_id to match. But not quite end to end: Tempo's generator labels its exemplars traceID (verified live), so the span-metric exemplars don't match that destination. Find the fix yourself and choose one: the datasource, Tempo, or both.
• Push vs pull. Prometheus scrapes (pull); OTLP pushes. With the Collector you can do either — receive OTLP from apps, then either expose a /metrics endpoint for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that the pull model is why Prometheus has service discovery and why up exists.

Resources

• Cloudflare — How Cloudflare runs Prometheus at scale (free article, ~20 min). Read the first half. Best cardinality explanation available.
• Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill, not a fact.
• The OneUptime APM article — now it's time. Take its spanmetrics / servicegraph connector config and its RED PromQL queries almost verbatim.
• OpenTelemetry docs → Python → Metrics; Semantic Conventions → Metrics.

Build

1. Extend common/telemetry.py with MeterProvider + OTLP metric exporter.
2. Custom application metrics, deliberately few:
   • ledgerline.transactions.total (Counter; labels: kind, status, currency)
   • ledgerline.transaction.duration (Histogram; labels: kind)
   • ledgerline.fraud.score (Histogram; no labels)
   • ledgerline.stream.lag (Observable Gauge; label: consumer_group)
   • ledgerline.db.pool.in_use (Observable UpDownCounter; label: service)
3. Get your custom metrics somewhere they'll actually land. Alloy rejects OTLP metrics (404 on /v1/metrics), so pick a path and implement it:
   • Platform grain: expose /metrics on each service and add scrape jobs to k8s/prometheus/prometheus.yml (the role's prometheus_extra_scrape_jobs), then kubectl apply -k k8s/. Prometheus scrapes your services directly. First decide how Prometheus on falken reaches czy: the Traefik override strips every service's host port (see phase-1.md). Note the irony and the lesson — you're back to a pull model and a second client library, which is exactly the fragmentation OTel exists to fix.
   • Match the diagram: add prometheus.scrape + prometheus.remote_write to Alloy so app metrics flow through the collector as the architecture drawing intends. This also closes a real diagram-vs-code gap in the repo.
   • Go all-OTLP: add a metrics output on otelcol.receiver.otlp, wire otelcol.exporter.prometheus → prometheus.remote_write. One OTLP path for all three signals. Do the first to get moving; do one of the others afterwards and write up which you'd choose for a real platform team and why. That write-up is a Phase 10 artefact.
4. Deliberately break it: add account_id as a label to ledgerline.transactions.total, run a 200-request burst, and watch the series count in Prometheus (count({__name__=~"ledgerline.*"})). Then remove it. Now you'll never forget.
5. Compare your hand-written duration histogram against the spanmetrics-generated one. Understand why they differ slightly and when you'd want each.
6. Second migration of the call-counting assertions. The three tests you moved to LogQL in Phase 4 can now count client spans in Tempo instead — a TraceQL query for sanctions client spans within a single trace answers "how many lookups did this one transaction make?" directly, which is the question the test was always really asking. Do it and compare all three generations: docker logs | grep → LogQL → TraceQL. Keep the diff; it's the clearest possible demonstration of what each signal is actually for, on a problem you already understand.

Done when: you have a RED view per service, a working service map, and you can click a latency spike through to a trace via an exemplar.