LedgerLine/docs/roadmap/phase-6.md

Phase 6 — Dashboards and alerts as code

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-5.md · Next: phase-7.md

Status: Done

Message: Done 2026-10-25. Both Done-when criteria verified live: Grafana's resources and its volume were deleted and kubectl apply -k k8s/ rebuilt all six dashboards and four datasources from the files (fresh PVC, every dashboard provisioned: true), and every dashboard's version history shows only the provisioner as author.

• Build steps done: 1, 2, 3, 4 (tracesToMetrics added; the tracesToLogsV2 question was not answered) and 5.
• Dashboards: observability/dashboards/generate.py (Foundation SDK) writes k8s/grafana/dashboards/*.json: red-<service> for the five HTTP services (gateway, ledger, fraud, sanctions, insights; audit and notifier serve no HTTP, so have none) and overview (service graph, stream lag, pool in use). No error-budget panel yet (Phase 8). The app exports no pool size, so the pool panel shows "in use", not saturation. The generator refuses to write while a FIXME remains and checks the total size against the ConfigMap limit.
• Alerts: Prometheus rule files, not Grafana alerting, delivering to Alertmanager (default receiver is still a no-op). LedgerlineTargetDown and LedgerlineHighErrorRatio in k8s/prometheus/rules/ledgerline.rules.yml, under an alerting contract (labels severity/team/service; annotations summary/description/runbook_url/dashboard) enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. A scratch alert was seen in Alertmanager; LedgerlineHighErrorRatio was seen pending on real traffic, but its firing and rendered annotations were not verified. The runbook URLs are a guessed Gitea-style path and returned 404 before the files were pushed.
• Not done: answering the tracesToLogsV2 / filterByTraceID question (the QUESTION comment is still in k8s/grafana/datasources.yml); clicking "Related metrics" on a span to confirm tracesToMetrics end to end; a real firing of either ledgerline alert; Grafana-managed alerting (Phase 8 decides).
• Carry forward:
  • The SDK's PyPI versions are <epoch>!<grafana-version>, and pip's default picks the oldest schema (10.1). Pin grafana-foundation-sdk==1769699379!11.6.0 (newest published is the 11.6 schema; Grafana 12.1.1 loads it).
  • Provisioned dashboards can't be deleted from the UI or API in Grafana 12.1.1 whatever disableDeletion says (the error is "provisioned dashboard cannot be deleted"). Removing the file removes the dashboard. The destroy test is kubectl kustomize k8s/grafana | ssh falken kubectl delete -n ledgerline-observability -f - then re-apply. Do not run kubectl delete -k k8s/: the root kustomization includes the Prometheus, Loki and Tempo PVCs and would delete all telemetry.
  • The configMap hash suffix rolls the pod on every change and leaves old hashed ConfigMaps behind (plain apply doesn't prune them).
  • Tempo's service map is built in the browser from Prometheus queries; the serviceMap query type returns "unsupported query type" through /api/ds/query, so that panel can only be checked by eye.
  • An unlabelled series renders as "Value" in a legend. The gateway's 404s have no http_route, so the rate panel shows one.
  • team on an alert is static per rule (who answers the class of alert), not per service. Phase 8's burn-rate rules belong in k8s/prometheus/rules/ under the same contract.

Time-box: 2 evenings. This is an explicit requirement from Babak: destroy and recreate in seconds.

Concepts

• Three levels of "as code", and you should know when each applies:
  1. File provisioning — YAML in provisioning/datasources/ and provisioning/dashboards/, plus dashboard JSON files mounted into the container. Grafana loads them at startup. Simplest, no state, no API tokens, perfect for a Compose demo. This alone satisfies the requirement.
  2. Grafana Foundation SDK — typed libraries (Go, Python, TypeScript, Java, PHP) that generate the dashboard JSON programmatically. Now a dashboard is a function: red_dashboard("ledger"), called seven times in a loop. Reviewable diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead of Grafonnet (which is legacy and unofficial).
  3. Terraform provider / Grafana Operator / Git Sync — for state-managed, multi-environment, or GitOps setups. Overkill here; the right answer later, when this lands on your Kubernetes cluster.
• Dashboards should answer questions, not display data. A good service dashboard answers, in order: is it up, is it serving, is it erroring, is it slow, and where is the pressure. Panels that nobody looks at during an incident are cost, not value.
• Alerts as code too. Grafana alert rules, contact points and notification policies all provision from YAML. And keep the alert definition next to the dashboard that explains it.

Resources

• Grafana docs → Observability as code (the index page — it lays out Foundation SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
• Grafana Foundation SDK repo → the Python examples, especially the red-method example.
• Udemy backbone: dashboard design sections (panel types, transformations, variables, dashboard best practice).

Build

Your platform provisions datasources only — there is no dashboards provider at all. So this phase is entirely yours, and the right home for it is inside the platform's own manifests (k8s/grafana/, the k3s port of the Ansible role), using the same mechanism as everything else.

1. Extend k8s/grafana/. Add a file-based dashboard provider (a provisioning/dashboards/ YAML pointing at a mounted directory) and the dashboard JSON. Ship both through configMapGenerator, exactly as datasources.yml is shipped today, and mount them in deployment.yaml. Two k8s-specific things to work out:
   • A ConfigMap is capped at about 1 MiB, and client-side kubectl apply also stores a copy in an annotation. Large generated dashboards push against that.
   • The configMap hash suffix rolls the Grafana pod on every change. Decide whether that's what you want.
2. Generate the JSON from code, don't hand-write it. Use the Grafana Foundation SDK (Python): one function producing a RED dashboard for a given service name, looped over the seven services; a second producing a system overview (service graph, stream lag, pool saturation, error-budget burn). Commit the generator and its output; k8s/grafana/ ships the output.
3. The demo is now kubectl apply -k k8s/. Delete every dashboard through the Grafana UI, re-apply (or go further: kubectl delete -k k8s/ and rebuild the whole platform), and watch them all come back. That is a strictly stronger answer to Babak's "destroy and recreate in seconds" than a local Makefile target, because it's the same command that builds the platform.
4. While you're in there, review the provisioned datasources and add what's missing: tracesToMetrics on Tempo (pointing at the Tempo-generated span metrics) is a worthwhile addition, and the existing tracesToLogsV2 has no tags mapping — understand why filterByTraceID: true alone is sufficient here, and when it wouldn't be.
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.

Done when: kubectl apply -k k8s/ restores every dashboard from scratch, and no dashboard has ever been saved through the Grafana UI.