Add project documentation and Claude instructions

Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.

docs/roadmap/phase-4.md

@@ -0,0 +1,126 @@
+# Phase 4 — Metrics
+
+*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
+
+**Status:** Not started
+
+**Message:** Not blocked technically (metrics don't strictly need tracing
+first), but sequenced after Phase 3 per the roadmap's ordering.
+
+**Time-box:** 2 evenings.
+
+### Concepts
+
+- **Instrument types.**
+  - **Counter** — monotonic, only increases. Requests, errors, messages consumed.
+  - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
+    connection-pool checkouts.
+  - **Histogram** — a distribution. Request latency, payload size. This is what
+    gives you percentiles.
+  - Synchronous instruments are recorded inline; **observable/asynchronous** ones are
+    read via a callback on collection (right for things you *poll*, like pool size).
+  - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
+    **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
+    nonsense. Use histograms.
+- **Cardinality — the one thing that will bite you.** Every unique combination of
+  metric name + label values is a separate time series in memory. Add `account_id`
+  as a label on a 800-account demo and you've made 800 series; do it in production
+  and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
+  ~10 000 distinct values in production, it does not belong on a metric.** Put it on
+  a span. Histograms multiply this (one series per bucket per label combination), so
+  don't over-bucket either.
+  - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
+    the concrete path), `http.status_code` class, `kind`, `currency`.
+  - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
+    raw URL, timestamp, error message.
+- **RED and USE.**
+  - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
+    per-service dashboard.
+  - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
+    connection pools, queues, disk. Saturation is the one everyone forgets and the
+    one that predicts collapse.
+  - You want both: RED tells you the service is slow, USE tells you which resource
+    is why.
+- **Span metrics — generate metrics from traces.** Two architectures exist for this,
+  and knowing both is the point:
+  - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
+    `servicegraph` connectors read spans in flight and emit RED metrics and edge
+    metrics. This is what most blog posts (including the OneUptime article) show.
+  - *In the trace store:* Tempo's **metrics-generator** does the same job after
+    ingestion, remote-writing the result to Prometheus. **This is what your platform
+    does** — `processors: [service-graphs, span-metrics, local-blocks]` with
+    `send_exemplars: true`, remote-writing to a Prometheus started with
+    `--web.enable-remote-write-receiver`.
+  - So RED metrics, the service map and exemplars **already exist** before you write
+    a single metric line. Do not add the collector connectors as well — you'd
+    duplicate every series under a second set of names. **Go find the real metric
+    names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
+    they will not match the article's.
+  - The trade-off worth being able to argue: collector-side generation is cheaper
+    and works before storage, but sees only what passes through that collector;
+    Tempo-side generation sees everything that landed, survives collector restarts,
+    and can be enabled per-tenant — at the cost of load on the trace store.
+- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
+  click a spike on a latency graph and land in the exact trace. On your platform
+  this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
+  and the Prometheus datasource declares `exemplarTraceIdDestinations` with
+  `name: trace_id`. If you emit your own histograms with exemplars, the label must
+  be `trace_id` to match.
+- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
+  can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
+  for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
+  the pull model is why Prometheus has service discovery and why `up` exists.
+
+### Resources
+
+- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
+  Read the first half. Best cardinality explanation available.
+- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
+  not a fact.
+- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
+  `servicegraph` connector config and its RED PromQL queries almost verbatim.
+- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
+
+### Build
+
+1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
+2. Custom application metrics, deliberately few:
+   - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
+   - `ledgerline.transaction.duration` (Histogram; labels: `kind`)
+   - `ledgerline.fraud.score` (Histogram; no labels)
+   - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
+   - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
+3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
+   metrics, so pick a path and implement it:
+   - *Platform grain:* expose `/metrics` on each service and add them to
+     `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
+     playbook. Prometheus scrapes your services directly. Note the irony and the
+     lesson — you're back to a pull model and a second client library, which is
+     exactly the fragmentation OTel exists to fix.
+   - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
+     Alloy so app metrics flow through the collector as the architecture drawing
+     intends. This also closes a real diagram-vs-code gap in the repo.
+   - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
+     `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
+     all three signals.
+   Do the first to get moving; do one of the others afterwards and write up which
+   you'd choose for a real platform team and why. That write-up is a Phase 9
+   artefact.
+4. Deliberately break it: add `account_id` as a label to
+   `ledgerline.transactions.total`, run a 200-request burst, and watch the series
+   count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
+   you'll never forget.
+5. Compare your hand-written duration histogram against the `spanmetrics`-generated
+   one. Understand why they differ slightly and when you'd want each.
+6. **Second migration of the call-counting assertions.** The three tests you moved
+   to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
+   TraceQL query for `sanctions` client spans within a single trace answers
+   "how many lookups did this one transaction make?" directly, which is the
+   question the test was always really asking. Do it and compare all three
+   generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
+   clearest possible demonstration of what each signal is actually for, on a
+   problem you already understand.
+
+**Done when:** you have a RED view per service, a working service map, and you can
+click a latency spike through to a trace via an exemplar.
+