LedgerLine/docs/roadmap/phase-5.md

Phase 5 — Metrics

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-4.md · Next: phase-6.md

Status: Not started

Message: Not blocked technically (metrics don't strictly need tracing first), but sequenced after Phase 4 per the roadmap's ordering.

Time-box: 2 evenings.

Concepts

• Instrument types.
  • Counter — monotonic, only increases. Requests, errors, messages consumed.
  • UpDownCounter / Gauge — can go both ways. Queue depth, in-flight requests, connection-pool checkouts.
  • Histogram — a distribution. Request latency, payload size. This is what gives you percentiles.
  • Synchronous instruments are recorded inline; observable/asynchronous ones are read via a callback on collection (right for things you poll, like pool size).
  • Avoid Prometheus Summaries: they compute quantiles client-side and therefore cannot be aggregated across instances. avg(p99_from_summary) is arithmetic nonsense. Use histograms.
• Cardinality — the one thing that will bite you. Every unique combination of metric name + label values is a separate time series in memory. Add account_id as a label on a 800-account demo and you've made 800 series; do it in production and you make millions and OOM Prometheus. Rule of thumb: if a label could exceed ~10 000 distinct values in production, it does not belong on a metric. Put it on a span. Histograms multiply this (one series per bucket per label combination), so don't over-bucket either.
  • Bounded and fine: service, http.route (the template, /accounts/{id}, not the concrete path), http.status_code class, kind, currency.
  • Unbounded and forbidden: account_id, transaction_id, merchant_name, raw URL, timestamp, error message.
• RED and USE.
  • RED (Rate, Errors, Duration) — for request-driven services. This is your per-service dashboard.
  • USE (Utilization, Saturation, Errors) — for resources: CPU, memory, connection pools, queues, disk. Saturation is the one everyone forgets and the one that predicts collapse.
  • You want both: RED tells you the service is slow, USE tells you which resource is why.
• Span metrics — generate metrics from traces. Two architectures exist for this, and knowing both is the point:
  • In the collector: the OpenTelemetry Collector's spanmetrics and servicegraph connectors read spans in flight and emit RED metrics and edge metrics. This is what most blog posts (including the OneUptime article) show.
  • In the trace store: Tempo's metrics-generator does the same job after ingestion, remote-writing the result to Prometheus. This is what your platform does — processors: [service-graphs, span-metrics, local-blocks] with send_exemplars: true, remote-writing to a Prometheus started with --web.enable-remote-write-receiver.
  • So RED metrics, the service map and exemplars already exist before you write a single metric line. Do not add the collector connectors as well — you'd duplicate every series under a second set of names. Go find the real metric names in Prometheus (/api/v1/label/__name__/values) before writing PromQL; they will not match the article's.
  • The trade-off worth being able to argue: collector-side generation is cheaper and works before storage, but sees only what passes through that collector; Tempo-side generation sees everything that landed, survives collector restarts, and can be enabled per-tenant — at the cost of load on the trace store.
• Exemplars. A histogram bucket can carry a sample trace ID. That's what lets you click a spike on a latency graph and land in the exact trace. On your platform this is wired: Tempo's generator sets send_exemplars: true, Prometheus stores them (--enable-feature=exemplar-storage, added in k8s/; the Ansible role lacked it), and the Prometheus datasource declares exemplarTraceIdDestinations with name: trace_id. If you emit your own histograms with exemplars, the label must be trace_id to match. But not quite end to end: Tempo's generator labels its exemplars traceID (verified live), so the span-metric exemplars don't match that destination. Find the fix yourself and choose one: the datasource, Tempo, or both.
• Push vs pull. Prometheus scrapes (pull); OTLP pushes. With the Collector you can do either — receive OTLP from apps, then either expose a /metrics endpoint for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that the pull model is why Prometheus has service discovery and why up exists.

Resources

• Cloudflare — How Cloudflare runs Prometheus at scale (free article, ~20 min). Read the first half. Best cardinality explanation available.
• Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill, not a fact.
• The OneUptime APM article — now it's time. Take its spanmetrics / servicegraph connector config and its RED PromQL queries almost verbatim.
• OpenTelemetry docs → Python → Metrics; Semantic Conventions → Metrics.

Build

1. Extend common/telemetry.py with MeterProvider + OTLP metric exporter.
2. Custom application metrics, deliberately few:
   • ledgerline.transactions.total (Counter; labels: kind, status, currency)
   • ledgerline.transaction.duration (Histogram; labels: kind)
   • ledgerline.fraud.score (Histogram; no labels)
   • ledgerline.stream.lag (Observable Gauge; label: consumer_group)
   • ledgerline.db.pool.in_use (Observable UpDownCounter; label: service)
3. Get your custom metrics somewhere they'll actually land. Alloy drops OTLP metrics, so pick a path and implement it:
   • Platform grain: expose /metrics on each service and add scrape jobs to k8s/prometheus/prometheus.yml (the role's prometheus_extra_scrape_jobs), then kubectl apply -k k8s/. Prometheus scrapes your services directly. First decide how Prometheus on falken reaches czy: the Traefik override strips every service's host port (see phase-1.md). Note the irony and the lesson — you're back to a pull model and a second client library, which is exactly the fragmentation OTel exists to fix.
   • Match the diagram: add prometheus.scrape + prometheus.remote_write to Alloy so app metrics flow through the collector as the architecture drawing intends. This also closes a real diagram-vs-code gap in the repo.
   • Go all-OTLP: add a metrics output on otelcol.receiver.otlp, wire otelcol.exporter.prometheus → prometheus.remote_write. One OTLP path for all three signals. Do the first to get moving; do one of the others afterwards and write up which you'd choose for a real platform team and why. That write-up is a Phase 10 artefact.
4. Deliberately break it: add account_id as a label to ledgerline.transactions.total, run a 200-request burst, and watch the series count in Prometheus (count({__name__=~"ledgerline.*"})). Then remove it. Now you'll never forget.
5. Compare your hand-written duration histogram against the spanmetrics-generated one. Understand why they differ slightly and when you'd want each.
6. Second migration of the call-counting assertions. The three tests you moved to LogQL in Phase 4 can now count client spans in Tempo instead — a TraceQL query for sanctions client spans within a single trace answers "how many lookups did this one transaction make?" directly, which is the question the test was always really asking. Do it and compare all three generations: docker logs | grep → LogQL → TraceQL. Keep the diff; it's the clearest possible demonstration of what each signal is actually for, on a problem you already understand.

Done when: you have a RED view per service, a working service map, and you can click a latency spike through to a trace via an exemplar.