# Phase 5 — Dashboards and alerts as code

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*

**Status:** Not started

**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
dashboarding.

**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
recreate in seconds.*

### Concepts

- Three levels of "as code", and you should know when each applies:
  1. **File provisioning** — YAML in `provisioning/datasources/` and
     `provisioning/dashboards/`, plus dashboard JSON files mounted into the
     container. Grafana loads them at startup. Simplest, no state, no API tokens,
     perfect for a Compose demo. **This alone satisfies the requirement.**
  2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
     PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
     function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
     diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
     of Grafonnet (which is legacy and unofficial).
  3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
     multi-environment, or GitOps setups. Overkill here; the right answer *later*,
     when this lands on your Kubernetes cluster.
- **Dashboards should answer questions, not display data.** A good service dashboard
  answers, in order: is it up, is it serving, is it erroring, is it slow, and where
  is the pressure. Panels that nobody looks at during an incident are cost, not value.
- **Alerts as code too.** Grafana alert rules, contact points and notification
  policies all provision from YAML. And keep the *alert definition* next to the
  dashboard that explains it.

### Resources

- Grafana docs → *Observability as code* (the index page — it lays out Foundation
  SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
  example.
- Udemy backbone: dashboard design sections (panel types, transformations,
  variables, dashboard best practice).

### Build

Your platform provisions **datasources only** — there is no dashboards provider at
all. So this phase is entirely yours, and the right home for it is inside the
Ansible role, using the same mechanism as everything else.

1. **Extend the role.** Add
   `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
   (a file-based dashboard provider pointing at a mounted directory), a
   `config/grafana/dashboards/` directory for the JSON, and the tasks to template
   and copy them. The Compose template already mounts
   `./config/grafana/provisioning` read-only, so the provider is picked up on
   restart.
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
   Foundation SDK (Python)**: one function producing a RED dashboard for a given
   service name, looped over the seven services; a second producing a system
   overview (service graph, stream lag, pool saturation, error-budget burn). Commit
   the generator *and* its output; the Ansible role ships the output.
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
   Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
   That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
   than a local Makefile target, because it's the same command that builds the
   platform.
4. While you're in there, review the provisioned datasources and add what's missing:
   `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
   worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
   understand why `filterByTraceID: true` alone is sufficient here, and when it
   wouldn't be.
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.

**Done when:** re-running the playbook restores every dashboard from scratch, and no
dashboard has ever been saved through the Grafana UI.