Add dashboards and alerts as code, mark Phase 6 done

Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.

Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-25 20:01:52 +03:30
co-authored by Claude Sonnet 5.5
parent b50f8c3b24
commit fef735c922
19 changed files with 1314 additions and 7 deletions
+53 -3
View File
@@ -2,10 +2,60 @@
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
**Status:** Not started
**Status:** Done
**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
dashboarding.
**Message:** Done 2026-10-25. Both Done-when criteria verified live: Grafana's
resources and its volume were deleted and `kubectl apply -k k8s/` rebuilt all
six dashboards and four datasources from the files (fresh PVC, every dashboard
`provisioned: true`), and every dashboard's version history shows only the
provisioner as author.
- **Build steps done:** 1, 2, 3, 4 (`tracesToMetrics` added; the
`tracesToLogsV2` question was not answered) and 5.
- **Dashboards:** `observability/dashboards/generate.py` (Foundation SDK)
writes `k8s/grafana/dashboards/*.json`: `red-<service>` for the five HTTP
services (gateway, ledger, fraud, sanctions, insights; audit and notifier
serve no HTTP, so have none) and `overview` (service graph, stream lag, pool
in use). No error-budget panel yet (Phase 8). The app exports no pool *size*,
so the pool panel shows "in use", not saturation. The generator refuses to
write while a `FIXME` remains and checks the total size against the
ConfigMap limit.
- **Alerts:** Prometheus rule files, not Grafana alerting, delivering to
Alertmanager (`default` receiver is still a no-op). `LedgerlineTargetDown`
and `LedgerlineHighErrorRatio` in `k8s/prometheus/rules/ledgerline.rules.yml`,
under an alerting contract (labels `severity`/`team`/`service`; annotations
`summary`/`description`/`runbook_url`/`dashboard`) enforced by
`observability/alerts/check_contract.py`, with runbooks in `docs/runbooks/`.
A scratch alert was seen in Alertmanager; `LedgerlineHighErrorRatio` was seen
`pending` on real traffic, but its firing and rendered annotations were not
verified. The runbook URLs are a guessed Gitea-style path and returned 404
before the files were pushed.
- **Not done:** answering the `tracesToLogsV2` / `filterByTraceID` question
(the `QUESTION` comment is still in `k8s/grafana/datasources.yml`); clicking
"Related metrics" on a span to confirm `tracesToMetrics` end to end; a real
firing of either `ledgerline` alert; Grafana-managed alerting (Phase 8
decides).
- **Carry forward:**
- The SDK's PyPI versions are `<epoch>!<grafana-version>`, and pip's default
picks the *oldest* schema (10.1). Pin `grafana-foundation-sdk==1769699379!11.6.0`
(newest published is the 11.6 schema; Grafana 12.1.1 loads it).
- Provisioned dashboards can't be deleted from the UI or API in Grafana
12.1.1 whatever `disableDeletion` says (the error is "provisioned dashboard
cannot be deleted"). Removing the *file* removes the dashboard. The destroy
test is `kubectl kustomize k8s/grafana | ssh falken kubectl delete -n
ledgerline-observability -f -` then re-apply. **Do not run `kubectl delete
-k k8s/`**: the root kustomization includes the Prometheus, Loki and Tempo
PVCs and would delete all telemetry.
- The configMap hash suffix rolls the pod on every change and leaves old
hashed ConfigMaps behind (plain `apply` doesn't prune them).
- Tempo's service map is built in the browser from Prometheus queries; the
`serviceMap` query type returns "unsupported query type" through
`/api/ds/query`, so that panel can only be checked by eye.
- An unlabelled series renders as "Value" in a legend. The gateway's 404s have
no `http_route`, so the rate panel shows one.
- `team` on an alert is static per rule (who answers the class of alert), not
per service. Phase 8's burn-rate rules belong in `k8s/prometheus/rules/`
under the same contract.
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
recreate in seconds.*