Movses AslanyanandClaude Sonnet 5.5
5e514ecf25
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
77 lines
4.2 KiB
Markdown
77 lines
4.2 KiB
Markdown
# Phase 6 — Dashboards and alerts as code
|
||
|
||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
|
||
|
||
**Status:** Not started
|
||
|
||
**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
|
||
dashboarding.
|
||
|
||
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
|
||
recreate in seconds.*
|
||
|
||
### Concepts
|
||
|
||
- Three levels of "as code", and you should know when each applies:
|
||
1. **File provisioning** — YAML in `provisioning/datasources/` and
|
||
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
|
||
container. Grafana loads them at startup. Simplest, no state, no API tokens,
|
||
perfect for a Compose demo. **This alone satisfies the requirement.**
|
||
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
|
||
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
|
||
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
|
||
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
|
||
of Grafonnet (which is legacy and unofficial).
|
||
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
|
||
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
|
||
when this lands on your Kubernetes cluster.
|
||
- **Dashboards should answer questions, not display data.** A good service dashboard
|
||
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
|
||
is the pressure. Panels that nobody looks at during an incident are cost, not value.
|
||
- **Alerts as code too.** Grafana alert rules, contact points and notification
|
||
policies all provision from YAML. And keep the *alert definition* next to the
|
||
dashboard that explains it.
|
||
|
||
### Resources
|
||
|
||
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
|
||
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
|
||
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
|
||
example.
|
||
- Udemy backbone: dashboard design sections (panel types, transformations,
|
||
variables, dashboard best practice).
|
||
|
||
### Build
|
||
|
||
Your platform provisions **datasources only** — there is no dashboards provider at
|
||
all. So this phase is entirely yours, and the right home for it is inside the
|
||
Ansible role, using the same mechanism as everything else.
|
||
|
||
1. **Extend the role.** Add
|
||
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
|
||
(a file-based dashboard provider pointing at a mounted directory), a
|
||
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
|
||
and copy them. The Compose template already mounts
|
||
`./config/grafana/provisioning` read-only, so the provider is picked up on
|
||
restart.
|
||
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
|
||
Foundation SDK (Python)**: one function producing a RED dashboard for a given
|
||
service name, looped over the seven services; a second producing a system
|
||
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
|
||
the generator *and* its output; the Ansible role ships the output.
|
||
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
|
||
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
|
||
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
|
||
than a local Makefile target, because it's the same command that builds the
|
||
platform.
|
||
4. While you're in there, review the provisioned datasources and add what's missing:
|
||
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
|
||
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
|
||
understand why `filterByTraceID: true` alone is sufficient here, and when it
|
||
wouldn't be.
|
||
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.
|
||
|
||
**Done when:** re-running the playbook restores every dashboard from scratch, and no
|
||
dashboard has ever been saved through the Grafana UI.
|
||
|