LedgerLine/docs/roadmap/phase-6.md

Phase 6 — Dashboards and alerts as code

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-5.md · Next: phase-7.md

Status: Not started

Message: Depends on Phases 3–5 producing traces, logs and metrics worth dashboarding.

Time-box: 2 evenings. This is an explicit requirement from Babak: destroy and recreate in seconds.

Concepts

• Three levels of "as code", and you should know when each applies:
  1. File provisioning — YAML in provisioning/datasources/ and provisioning/dashboards/, plus dashboard JSON files mounted into the container. Grafana loads them at startup. Simplest, no state, no API tokens, perfect for a Compose demo. This alone satisfies the requirement.
  2. Grafana Foundation SDK — typed libraries (Go, Python, TypeScript, Java, PHP) that generate the dashboard JSON programmatically. Now a dashboard is a function: red_dashboard("ledger"), called seven times in a loop. Reviewable diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead of Grafonnet (which is legacy and unofficial).
  3. Terraform provider / Grafana Operator / Git Sync — for state-managed, multi-environment, or GitOps setups. Overkill here; the right answer later, when this lands on your Kubernetes cluster.
• Dashboards should answer questions, not display data. A good service dashboard answers, in order: is it up, is it serving, is it erroring, is it slow, and where is the pressure. Panels that nobody looks at during an incident are cost, not value.
• Alerts as code too. Grafana alert rules, contact points and notification policies all provision from YAML. And keep the alert definition next to the dashboard that explains it.

Resources

• Grafana docs → Observability as code (the index page — it lays out Foundation SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
• Grafana Foundation SDK repo → the Python examples, especially the red-method example.
• Udemy backbone: dashboard design sections (panel types, transformations, variables, dashboard best practice).

Build

Your platform provisions datasources only — there is no dashboards provider at all. So this phase is entirely yours, and the right home for it is inside the platform's own manifests (k8s/grafana/, the k3s port of the Ansible role), using the same mechanism as everything else.

1. Extend k8s/grafana/. Add a file-based dashboard provider (a provisioning/dashboards/ YAML pointing at a mounted directory) and the dashboard JSON. Ship both through configMapGenerator, exactly as datasources.yml is shipped today, and mount them in deployment.yaml. Two k8s-specific things to work out:
   • A ConfigMap is capped at about 1 MiB, and client-side kubectl apply also stores a copy in an annotation. Large generated dashboards push against that.
   • The configMap hash suffix rolls the Grafana pod on every change. Decide whether that's what you want.
2. Generate the JSON from code, don't hand-write it. Use the Grafana Foundation SDK (Python): one function producing a RED dashboard for a given service name, looped over the seven services; a second producing a system overview (service graph, stream lag, pool saturation, error-budget burn). Commit the generator and its output; k8s/grafana/ ships the output.
3. The demo is now kubectl apply -k k8s/. Delete every dashboard through the Grafana UI, re-apply (or go further: kubectl delete -k k8s/ and rebuild the whole platform), and watch them all come back. That is a strictly stronger answer to Babak's "destroy and recreate in seconds" than a local Makefile target, because it's the same command that builds the platform.
4. While you're in there, review the provisioned datasources and add what's missing: tracesToMetrics on Tempo (pointing at the Tempo-generated span metrics) is a worthwhile addition, and the existing tracesToLogsV2 has no tags mapping — understand why filterByTraceID: true alone is sufficient here, and when it wouldn't be.
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.

Done when: kubectl apply -k k8s/ restores every dashboard from scratch, and no dashboard has ever been saved through the Grafana UI.