Add dashboards and alerts as code, mark Phase 6 done

Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.

Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/roadmap/phase-6.md

@@ -2,10 +2,60 @@
 
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
 
-**Status:** Not started
+**Status:** Done
 
-**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
-dashboarding.
+**Message:** Done 2026-10-25. Both Done-when criteria verified live: Grafana's
+resources and its volume were deleted and `kubectl apply -k k8s/` rebuilt all
+six dashboards and four datasources from the files (fresh PVC, every dashboard
+`provisioned: true`), and every dashboard's version history shows only the
+provisioner as author.
+
+- **Build steps done:** 1, 2, 3, 4 (`tracesToMetrics` added; the
+  `tracesToLogsV2` question was not answered) and 5.
+- **Dashboards:** `observability/dashboards/generate.py` (Foundation SDK)
+  writes `k8s/grafana/dashboards/*.json`: `red-<service>` for the five HTTP
+  services (gateway, ledger, fraud, sanctions, insights; audit and notifier
+  serve no HTTP, so have none) and `overview` (service graph, stream lag, pool
+  in use). No error-budget panel yet (Phase 8). The app exports no pool *size*,
+  so the pool panel shows "in use", not saturation. The generator refuses to
+  write while a `FIXME` remains and checks the total size against the
+  ConfigMap limit.
+- **Alerts:** Prometheus rule files, not Grafana alerting, delivering to
+  Alertmanager (`default` receiver is still a no-op). `LedgerlineTargetDown`
+  and `LedgerlineHighErrorRatio` in `k8s/prometheus/rules/ledgerline.rules.yml`,
+  under an alerting contract (labels `severity`/`team`/`service`; annotations
+  `summary`/`description`/`runbook_url`/`dashboard`) enforced by
+  `observability/alerts/check_contract.py`, with runbooks in `docs/runbooks/`.
+  A scratch alert was seen in Alertmanager; `LedgerlineHighErrorRatio` was seen
+  `pending` on real traffic, but its firing and rendered annotations were not
+  verified. The runbook URLs are a guessed Gitea-style path and returned 404
+  before the files were pushed.
+- **Not done:** answering the `tracesToLogsV2` / `filterByTraceID` question
+  (the `QUESTION` comment is still in `k8s/grafana/datasources.yml`); clicking
+  "Related metrics" on a span to confirm `tracesToMetrics` end to end; a real
+  firing of either `ledgerline` alert; Grafana-managed alerting (Phase 8
+  decides).
+- **Carry forward:**
+  - The SDK's PyPI versions are `<epoch>!<grafana-version>`, and pip's default
+    picks the *oldest* schema (10.1). Pin `grafana-foundation-sdk==1769699379!11.6.0`
+    (newest published is the 11.6 schema; Grafana 12.1.1 loads it).
+  - Provisioned dashboards can't be deleted from the UI or API in Grafana
+    12.1.1 whatever `disableDeletion` says (the error is "provisioned dashboard
+    cannot be deleted"). Removing the *file* removes the dashboard. The destroy
+    test is `kubectl kustomize k8s/grafana | ssh falken kubectl delete -n
+    ledgerline-observability -f -` then re-apply. **Do not run `kubectl delete
+    -k k8s/`**: the root kustomization includes the Prometheus, Loki and Tempo
+    PVCs and would delete all telemetry.
+  - The configMap hash suffix rolls the pod on every change and leaves old
+    hashed ConfigMaps behind (plain `apply` doesn't prune them).
+  - Tempo's service map is built in the browser from Prometheus queries; the
+    `serviceMap` query type returns "unsupported query type" through
+    `/api/ds/query`, so that panel can only be checked by eye.
+  - An unlabelled series renders as "Value" in a legend. The gateway's 404s have
+    no `http_route`, so the rate panel shows one.
+  - `team` on an alert is static per rule (who answers the class of alert), not
+    per service. Phase 8's burn-rate rules belong in `k8s/prometheus/rules/`
+    under the same contract.
 
 **Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
 recreate in seconds.*

docs/roadmap/summary.md

@@ -21,7 +21,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
 | 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
 | 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
 | 5 | [`phase-5.md`](phase-5.md) | Done | Metrics | 2 evenings | RED per service, service map, exemplars |
-| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
+| 6 | [`phase-6.md`](phase-6.md) | Done | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
 | 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
 | 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
 | 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
@@ -56,7 +56,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
   `phase-5.md` (second metrics path and the TraceQL test migration not done).
 - **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
   Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
-  "destroy and recreate in seconds" is a real, repeatable command.
+  "destroy and recreate in seconds" is a real, repeatable command. **Done** —
+  see `phase-6.md` (alerts as Prometheus rules under a contract; the
+  `tracesToLogsV2` question and a real alert firing not done).
 - **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
   be averaged, what coordinated omission is, and drive Ledgerline with
   arrival-rate executors to get a clean before/after load profile.