LedgerLine/docs/roadmap/phase-5.md

Phase 5 — Dashboards and alerts as code

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-4.md · Next: phase-6.md

Status: Not started

Message: Depends on Phases 2–4 producing traces, logs and metrics worth dashboarding.

Time-box: 2 evenings. This is an explicit requirement from Babak: destroy and recreate in seconds.

Concepts

• Three levels of "as code", and you should know when each applies:
  1. File provisioning — YAML in provisioning/datasources/ and provisioning/dashboards/, plus dashboard JSON files mounted into the container. Grafana loads them at startup. Simplest, no state, no API tokens, perfect for a Compose demo. This alone satisfies the requirement.
  2. Grafana Foundation SDK — typed libraries (Go, Python, TypeScript, Java, PHP) that generate the dashboard JSON programmatically. Now a dashboard is a function: red_dashboard("ledger"), called seven times in a loop. Reviewable diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead of Grafonnet (which is legacy and unofficial).
  3. Terraform provider / Grafana Operator / Git Sync — for state-managed, multi-environment, or GitOps setups. Overkill here; the right answer later, when this lands on your Kubernetes cluster.
• Dashboards should answer questions, not display data. A good service dashboard answers, in order: is it up, is it serving, is it erroring, is it slow, and where is the pressure. Panels that nobody looks at during an incident are cost, not value.
• Alerts as code too. Grafana alert rules, contact points and notification policies all provision from YAML. And keep the alert definition next to the dashboard that explains it.

Resources

• Grafana docs → Observability as code (the index page — it lays out Foundation SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
• Grafana Foundation SDK repo → the Python examples, especially the red-method example.
• Udemy backbone: dashboard design sections (panel types, transformations, variables, dashboard best practice).

Build

Your platform provisions datasources only — there is no dashboards provider at all. So this phase is entirely yours, and the right home for it is inside the Ansible role, using the same mechanism as everything else.

1. Extend the role. Add roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2 (a file-based dashboard provider pointing at a mounted directory), a config/grafana/dashboards/ directory for the JSON, and the tasks to template and copy them. The Compose template already mounts ./config/grafana/provisioning read-only, so the provider is picked up on restart.
2. Generate the JSON from code, don't hand-write it. Use the Grafana Foundation SDK (Python): one function producing a RED dashboard for a given service name, looped over the seven services; a second producing a system overview (service graph, stream lag, pool saturation, error-budget burn). Commit the generator and its output; the Ansible role ships the output.
3. The demo is now ansible-playbook. Delete every dashboard through the Grafana UI, re-run playbooks/grafana-stack.yml, and watch them all come back. That is a strictly stronger answer to Babak's "destroy and recreate in seconds" than a local Makefile target, because it's the same command that builds the platform.
4. While you're in there, review the provisioned datasources and add what's missing: tracesToMetrics on Tempo (pointing at the Tempo-generated span metrics) is a worthwhile addition, and the existing tracesToLogsV2 has no tags mapping — understand why filterByTraceID: true alone is sufficient here, and when it wouldn't be.
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.

Done when: re-running the playbook restores every dashboard from scratch, and no dashboard has ever been saved through the Grafana UI.