LedgerLine/docs/roadmap/phase-6.md

# Phase 6 — Dashboards and alerts as code

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*

**Status:** Done

**Message:** Done 2026-10-25. Both Done-when criteria verified live: Grafana's
resources and its volume were deleted and `kubectl apply -k k8s/` rebuilt all
six dashboards and four datasources from the files (fresh PVC, every dashboard
`provisioned: true`), and every dashboard's version history shows only the
provisioner as author.

- **Build steps done:** 1, 2, 3, 4 (`tracesToMetrics` added; the
  `tracesToLogsV2` question was not answered) and 5.
- **Dashboards:** `observability/dashboards/generate.py` (Foundation SDK)
  writes `k8s/grafana/dashboards/*.json`: `red-<service>` for the five HTTP
  services (gateway, ledger, fraud, sanctions, insights; audit and notifier
  serve no HTTP, so have none) and `overview` (service graph, stream lag, pool
  in use). No error-budget panel yet (Phase 8). The app exports no pool *size*,
  so the pool panel shows "in use", not saturation. The generator refuses to
  write while a `FIXME` remains and checks the total size against the
  ConfigMap limit.
- **Alerts:** Prometheus rule files, not Grafana alerting, delivering to
  Alertmanager (`default` receiver is still a no-op). `LedgerlineTargetDown`
  and `LedgerlineHighErrorRatio` in `k8s/prometheus/rules/ledgerline.rules.yml`,
  under an alerting contract (labels `severity`/`team`/`service`; annotations
  `summary`/`description`/`runbook_url`/`dashboard`) enforced by
  `observability/alerts/check_contract.py`, with runbooks in `docs/runbooks/`.
  A scratch alert was seen in Alertmanager; `LedgerlineHighErrorRatio` was seen
  `pending` on real traffic, but its firing and rendered annotations were not
  verified. The runbook URLs are a guessed Gitea-style path and returned 404
  before the files were pushed.
- **Not done:** answering the `tracesToLogsV2` / `filterByTraceID` question
  (the `QUESTION` comment is still in `k8s/grafana/datasources.yml`); clicking
  "Related metrics" on a span to confirm `tracesToMetrics` end to end; a real
  firing of either `ledgerline` alert; Grafana-managed alerting (Phase 8
  decides).
- **Carry forward:**
  - The SDK's PyPI versions are `<epoch>!<grafana-version>`, and pip's default
    picks the *oldest* schema (10.1). Pin `grafana-foundation-sdk==1769699379!11.6.0`
    (newest published is the 11.6 schema; Grafana 12.1.1 loads it).
  - Provisioned dashboards can't be deleted from the UI or API in Grafana
    12.1.1 whatever `disableDeletion` says (the error is "provisioned dashboard
    cannot be deleted"). Removing the *file* removes the dashboard. The destroy
    test is `kubectl kustomize k8s/grafana | ssh falken kubectl delete -n
    ledgerline-observability -f -` then re-apply. **Do not run `kubectl delete
    -k k8s/`**: the root kustomization includes the Prometheus, Loki and Tempo
    PVCs and would delete all telemetry.
  - The configMap hash suffix rolls the pod on every change and leaves old
    hashed ConfigMaps behind (plain `apply` doesn't prune them).
  - Tempo's service map is built in the browser from Prometheus queries; the
    `serviceMap` query type returns "unsupported query type" through
    `/api/ds/query`, so that panel can only be checked by eye.
  - An unlabelled series renders as "Value" in a legend. The gateway's 404s have
    no `http_route`, so the rate panel shows one.
  - `team` on an alert is static per rule (who answers the class of alert), not
    per service. Phase 8's burn-rate rules belong in `k8s/prometheus/rules/`
    under the same contract.

**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
recreate in seconds.*

### Concepts

- Three levels of "as code", and you should know when each applies:
  1. **File provisioning** — YAML in `provisioning/datasources/` and
     `provisioning/dashboards/`, plus dashboard JSON files mounted into the
     container. Grafana loads them at startup. Simplest, no state, no API tokens,
     perfect for a Compose demo. **This alone satisfies the requirement.**
  2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
     PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
     function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
     diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
     of Grafonnet (which is legacy and unofficial).
  3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
     multi-environment, or GitOps setups. Overkill here; the right answer *later*,
     when this lands on your Kubernetes cluster.
- **Dashboards should answer questions, not display data.** A good service dashboard
  answers, in order: is it up, is it serving, is it erroring, is it slow, and where
  is the pressure. Panels that nobody looks at during an incident are cost, not value.
- **Alerts as code too.** Grafana alert rules, contact points and notification
  policies all provision from YAML. And keep the *alert definition* next to the
  dashboard that explains it.

### Resources

- Grafana docs → *Observability as code* (the index page — it lays out Foundation
  SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
  example.
- Udemy backbone: dashboard design sections (panel types, transformations,
  variables, dashboard best practice).

### Build

Your platform provisions **datasources only** — there is no dashboards provider at
all. So this phase is entirely yours, and the right home for it is inside the
platform's own manifests (`k8s/grafana/`, the k3s port of the Ansible role),
using the same mechanism as everything else.

1. **Extend `k8s/grafana/`.** Add a file-based dashboard provider (a
   `provisioning/dashboards/` YAML pointing at a mounted directory) and the
   dashboard JSON. Ship both through `configMapGenerator`, exactly as
   `datasources.yml` is shipped today, and mount them in `deployment.yaml`.
   Two k8s-specific things to work out:
   - A ConfigMap is capped at about 1 MiB, and client-side `kubectl apply`
     also stores a copy in an annotation. Large generated dashboards push
     against that.
   - The configMap hash suffix rolls the Grafana pod on every change.
     Decide whether that's what you want.
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
   Foundation SDK (Python)**: one function producing a RED dashboard for a given
   service name, looped over the seven services; a second producing a system
   overview (service graph, stream lag, pool saturation, error-budget burn). Commit
   the generator *and* its output; `k8s/grafana/` ships the output.
3. **The demo is now `kubectl apply -k k8s/`.** Delete every dashboard through
   the Grafana UI, re-apply (or go further: `kubectl delete -k k8s/` and rebuild
   the whole platform), and watch them all come back.
   That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
   than a local Makefile target, because it's the same command that builds the
   platform.
4. While you're in there, review the provisioned datasources and add what's missing:
   `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
   worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
   understand why `filterByTraceID: true` alone is sufficient here, and when it
   wouldn't be.
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.

**Done when:** `kubectl apply -k k8s/` restores every dashboard from scratch, and no
dashboard has ever been saved through the Grafana UI.