Add dashboards and alerts as code, mark Phase 6 done

Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.

Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/roadmap/phase-6.md

@@ -2,10 +2,60 @@
 
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
 
-**Status:** Not started
+**Status:** Done
 
-**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
+**Message:** Done 2026-10-25. Both Done-when criteria verified live: Grafana's
-dashboarding.
+resources and its volume were deleted and `kubectl apply -k k8s/` rebuilt all
+six dashboards and four datasources from the files (fresh PVC, every dashboard
+`provisioned: true`), and every dashboard's version history shows only the
+provisioner as author.
+
+- **Build steps done:** 1, 2, 3, 4 (`tracesToMetrics` added; the
+  `tracesToLogsV2` question was not answered) and 5.
+- **Dashboards:** `observability/dashboards/generate.py` (Foundation SDK)
+  writes `k8s/grafana/dashboards/*.json`: `red-<service>` for the five HTTP
+  services (gateway, ledger, fraud, sanctions, insights; audit and notifier
+  serve no HTTP, so have none) and `overview` (service graph, stream lag, pool
+  in use). No error-budget panel yet (Phase 8). The app exports no pool *size*,
+  so the pool panel shows "in use", not saturation. The generator refuses to
+  write while a `FIXME` remains and checks the total size against the
+  ConfigMap limit.
+- **Alerts:** Prometheus rule files, not Grafana alerting, delivering to
+  Alertmanager (`default` receiver is still a no-op). `LedgerlineTargetDown`
+  and `LedgerlineHighErrorRatio` in `k8s/prometheus/rules/ledgerline.rules.yml`,
+  under an alerting contract (labels `severity`/`team`/`service`; annotations
+  `summary`/`description`/`runbook_url`/`dashboard`) enforced by
+  `observability/alerts/check_contract.py`, with runbooks in `docs/runbooks/`.
+  A scratch alert was seen in Alertmanager; `LedgerlineHighErrorRatio` was seen
+  `pending` on real traffic, but its firing and rendered annotations were not
+  verified. The runbook URLs are a guessed Gitea-style path and returned 404
+  before the files were pushed.
+- **Not done:** answering the `tracesToLogsV2` / `filterByTraceID` question
+  (the `QUESTION` comment is still in `k8s/grafana/datasources.yml`); clicking
+  "Related metrics" on a span to confirm `tracesToMetrics` end to end; a real
+  firing of either `ledgerline` alert; Grafana-managed alerting (Phase 8
+  decides).
+- **Carry forward:**
+  - The SDK's PyPI versions are `<epoch>!<grafana-version>`, and pip's default
+    picks the *oldest* schema (10.1). Pin `grafana-foundation-sdk==1769699379!11.6.0`
+    (newest published is the 11.6 schema; Grafana 12.1.1 loads it).
+  - Provisioned dashboards can't be deleted from the UI or API in Grafana
+    12.1.1 whatever `disableDeletion` says (the error is "provisioned dashboard
+    cannot be deleted"). Removing the *file* removes the dashboard. The destroy
+    test is `kubectl kustomize k8s/grafana | ssh falken kubectl delete -n
+    ledgerline-observability -f -` then re-apply. **Do not run `kubectl delete
+    -k k8s/`**: the root kustomization includes the Prometheus, Loki and Tempo
+    PVCs and would delete all telemetry.
+  - The configMap hash suffix rolls the pod on every change and leaves old
+    hashed ConfigMaps behind (plain `apply` doesn't prune them).
+  - Tempo's service map is built in the browser from Prometheus queries; the
+    `serviceMap` query type returns "unsupported query type" through
+    `/api/ds/query`, so that panel can only be checked by eye.
+  - An unlabelled series renders as "Value" in a legend. The gateway's 404s have
+    no `http_route`, so the rate panel shows one.
+  - `team` on an alert is static per rule (who answers the class of alert), not
+    per service. Phase 8's burn-rate rules belong in `k8s/prometheus/rules/`
+    under the same contract.
 
 **Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
 recreate in seconds.*

docs/roadmap/summary.md

@@ -21,7 +21,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
 | 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
 | 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
 | 5 | [`phase-5.md`](phase-5.md) | Done | Metrics | 2 evenings | RED per service, service map, exemplars |
-| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
+| 6 | [`phase-6.md`](phase-6.md) | Done | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
 | 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
 | 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
 | 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
@@ -56,7 +56,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
   `phase-5.md` (second metrics path and the TraceQL test migration not done).
 - **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
   Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
-  "destroy and recreate in seconds" is a real, repeatable command.
+  "destroy and recreate in seconds" is a real, repeatable command. **Done** —
+  see `phase-6.md` (alerts as Prometheus rules under a contract; the
+  `tracesToLogsV2` question and a real alert firing not done).
 - **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
   be averaged, what coordinated omission is, and drive Ledgerline with
   arrival-rate executors to get a clean before/after load profile.

docs/runbooks/ledgerline-high-error-ratio.md

@@ -0,0 +1,27 @@
+# LedgerlineHighErrorRatio
+
+**Fires when:** a service's 5xx ratio is above 5% for 5 minutes, with more than
+0.05 requests/s (so it is not one failed request on an idle service).
+**Severity:** warning · **Team:** payments
+
+## What it means
+
+A meaningful share of the requests this service serves are failing with a server
+error. 4xx responses (for example a 404) are deliberately not counted.
+
+## First five minutes
+
+1. **Look at the `RED: <service>` dashboard** (linked in the alert). Is the error
+   ratio still elevated, or already back to normal? Did the request rate or the
+   latency change at the same time?
+2. **Find a failing request.** In Tempo, search for the service with an error
+   status, or in Loki: `{service_name="<service>"} | json | level="ERROR"`.
+3. **Follow one trace.** The `trace_id` in the log line opens the whole request
+   across services. Look for the first span in the trace with an error status.
+4. **Is it only this service?** The System overview's service graph shows whether
+   its callers or callees are failing as well.
+
+## Escalate when
+
+The ratio is still rising after ten minutes, or the failing requests are on the
+critical path (transactions), not a side route.

docs/runbooks/ledgerline-target-down.md

@@ -0,0 +1,29 @@
+# LedgerlineTargetDown
+
+**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes.
+**Severity:** critical · **Team:** platform
+
+## What it means
+
+Prometheus scrapes all seven services through one address,
+`https://ledgerline.vione.cloud/metrics/<service>` (czy's Traefik, basic auth).
+"Down" means that scrape failed. It does **not** by itself prove the service is
+down: the service, the proxy, or the credentials can each cause it.
+
+## First five minutes
+
+1. **How many targets?** Open the alert in Alertmanager. One service alerting is
+   likely that service. All seven at once points at the shared path (Traefik on
+   czy, DNS, the scrape credentials), not at the services.
+2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and
+   read its `Last error`.
+3. **Is the container running?** `ssh czy docker compose ps` for the service.
+4. **What did it last say?** `ssh czy docker logs ledgerline-<service>-1`, or in
+   Loki: `{service_name="<service>"}`.
+5. **Is traffic still flowing?** Open the `RED: <service>` dashboard. Traces and
+   logs can still be arriving while only the metrics scrape is broken.
+
+## Escalate when
+
+The shared path is down (all seven targets), or the service is down and not
+recovering after a restart.

k8s/grafana/dashboards-provider.yml

@@ -0,0 +1,13 @@
+apiVersion: 1
+
+providers:
+  - name: ledgerline
+    orgId: 1
+    type: file
+    folder: Ledgerline
+    options:
+      path: /etc/grafana/dashboards/
+    allowUiUpdates: false
+    disableDeletion: false
+    updateIntervalSeconds: 15
+    foldersFromFilesStructure: false

k8s/grafana/dashboards/overview.json

@@ -0,0 +1,115 @@
+{
+  "annotations": {},
+  "editable": false,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "panels": [
+    {
+      "datasource": {
+        "type": "tempo",
+        "uid": "tempo"
+      },
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "tempo",
+            "uid": "tempo"
+          },
+          "filters": [],
+          "queryType": "serviceMap",
+          "refId": "graph"
+        }
+      ],
+      "title": "Service graph",
+      "transparent": false,
+      "type": "nodeGraph"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 12
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "ledgerline_stream_lag",
+          "legendFormat": "{{consumer_group}}",
+          "refId": "lag"
+        }
+      ],
+      "title": "Stream consumer lag",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 12
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (service, pool) (ledgerline_db_pool_in_use)",
+          "legendFormat": "{{service}} / {{pool}}",
+          "refId": "pool"
+        }
+      ],
+      "title": "DB pool connections in use",
+      "transparent": false,
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "15s",
+  "schemaVersion": 41,
+  "tags": [
+    "generated",
+    "overview"
+  ],
+  "templating": {},
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "System overview",
+  "uid": "overview"
+}

k8s/grafana/dashboards/red-fraud.json

@@ -0,0 +1,140 @@
+{
+  "annotations": {},
+  "editable": false,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"fraud\"}[5m]))",
+          "legendFormat": "{{http_route}}",
+          "refId": "rate"
+        }
+      ],
+      "title": "Request rate",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "min": 0,
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"fraud\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"fraud\"}[5m]))",
+          "legendFormat": "5xx ratio",
+          "refId": "errors"
+        }
+      ],
+      "title": "Error ratio",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"fraud\"}[5m])))",
+          "legendFormat": "p50",
+          "refId": "p50"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"fraud\"}[5m])))",
+          "legendFormat": "p95",
+          "refId": "p95"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"fraud\"}[5m])))",
+          "legendFormat": "p99",
+          "refId": "p99"
+        }
+      ],
+      "title": "Request duration",
+      "transparent": false,
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "15s",
+  "schemaVersion": 41,
+  "tags": [
+    "generated",
+    "red"
+  ],
+  "templating": {},
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "RED: fraud",
+  "uid": "red-fraud"
+}

k8s/grafana/dashboards/red-gateway.json

@@ -0,0 +1,140 @@
+{
+  "annotations": {},
+  "editable": false,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"gateway\"}[5m]))",
+          "legendFormat": "{{http_route}}",
+          "refId": "rate"
+        }
+      ],
+      "title": "Request rate",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "min": 0,
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"gateway\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"gateway\"}[5m]))",
+          "legendFormat": "5xx ratio",
+          "refId": "errors"
+        }
+      ],
+      "title": "Error ratio",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"gateway\"}[5m])))",
+          "legendFormat": "p50",
+          "refId": "p50"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"gateway\"}[5m])))",
+          "legendFormat": "p95",
+          "refId": "p95"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"gateway\"}[5m])))",
+          "legendFormat": "p99",
+          "refId": "p99"
+        }
+      ],
+      "title": "Request duration",
+      "transparent": false,
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "15s",
+  "schemaVersion": 41,
+  "tags": [
+    "generated",
+    "red"
+  ],
+  "templating": {},
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "RED: gateway",
+  "uid": "red-gateway"
+}

k8s/grafana/dashboards/red-insights.json

@@ -0,0 +1,140 @@
+{
+  "annotations": {},
+  "editable": false,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"insights\"}[5m]))",
+          "legendFormat": "{{http_route}}",
+          "refId": "rate"
+        }
+      ],
+      "title": "Request rate",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "min": 0,
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"insights\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"insights\"}[5m]))",
+          "legendFormat": "5xx ratio",
+          "refId": "errors"
+        }
+      ],
+      "title": "Error ratio",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"insights\"}[5m])))",
+          "legendFormat": "p50",
+          "refId": "p50"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"insights\"}[5m])))",
+          "legendFormat": "p95",
+          "refId": "p95"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"insights\"}[5m])))",
+          "legendFormat": "p99",
+          "refId": "p99"
+        }
+      ],
+      "title": "Request duration",
+      "transparent": false,
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "15s",
+  "schemaVersion": 41,
+  "tags": [
+    "generated",
+    "red"
+  ],
+  "templating": {},
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "RED: insights",
+  "uid": "red-insights"
+}

k8s/grafana/dashboards/red-ledger.json

@@ -0,0 +1,140 @@
+{
+  "annotations": {},
+  "editable": false,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"ledger\"}[5m]))",
+          "legendFormat": "{{http_route}}",
+          "refId": "rate"
+        }
+      ],
+      "title": "Request rate",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "min": 0,
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"ledger\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"ledger\"}[5m]))",
+          "legendFormat": "5xx ratio",
+          "refId": "errors"
+        }
+      ],
+      "title": "Error ratio",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"ledger\"}[5m])))",
+          "legendFormat": "p50",
+          "refId": "p50"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"ledger\"}[5m])))",
+          "legendFormat": "p95",
+          "refId": "p95"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"ledger\"}[5m])))",
+          "legendFormat": "p99",
+          "refId": "p99"
+        }
+      ],
+      "title": "Request duration",
+      "transparent": false,
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "15s",
+  "schemaVersion": 41,
+  "tags": [
+    "generated",
+    "red"
+  ],
+  "templating": {},
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "RED: ledger",
+  "uid": "red-ledger"
+}

k8s/grafana/dashboards/red-sanctions.json

@@ -0,0 +1,140 @@
+{
+  "annotations": {},
+  "editable": false,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"sanctions\"}[5m]))",
+          "legendFormat": "{{http_route}}",
+          "refId": "rate"
+        }
+      ],
+      "title": "Request rate",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "min": 0,
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"sanctions\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"sanctions\"}[5m]))",
+          "legendFormat": "5xx ratio",
+          "refId": "errors"
+        }
+      ],
+      "title": "Error ratio",
+      "transparent": false,
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"sanctions\"}[5m])))",
+          "legendFormat": "p50",
+          "refId": "p50"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"sanctions\"}[5m])))",
+          "legendFormat": "p95",
+          "refId": "p95"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"sanctions\"}[5m])))",
+          "legendFormat": "p99",
+          "refId": "p99"
+        }
+      ],
+      "title": "Request duration",
+      "transparent": false,
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "15s",
+  "schemaVersion": 41,
+  "tags": [
+    "generated",
+    "red"
+  ],
+  "templating": {},
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "RED: sanctions",
+  "uid": "red-sanctions"
+}

k8s/grafana/datasources.yml

@@ -38,6 +38,22 @@ datasources:
         filterByTraceID: true
         spanStartTimeShift: -1m
         spanEndTimeShift: 1m
+        # QUESTION (answer in your own words; verify by reading the LogQL that
+        # "Logs for this span" generates): why is `filterByTraceID: true` enough
+        # with no `tags` mapping here, and when would it not be?
+      tracesToMetrics:
+        datasourceUid: prometheus
+        # Wider than the span: rate() needs a few scrape intervals of data.
+        spanStartTimeShift: -2m
+        spanEndTimeShift: 2m
+        # span attribute -> Prometheus label on traces_spanmetrics_*
+        tags:
+          - key: service.name
+            value: service
+        queries:
+          # `$$` is the provisioning-file escape for a literal `$`.
+          - name: Request rate
+            query: sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))
       serviceMap:
         datasourceUid: prometheus
       nodeGraph:

k8s/grafana/deployment.yaml

@@ -51,6 +51,10 @@ spec:
               mountPath: /var/lib/grafana
             - name: datasources
               mountPath: /etc/grafana/provisioning/datasources
+            - name: dashboards-provider
+              mountPath: /etc/grafana/provisioning/dashboards
+            - name: dashboards
+              mountPath: /etc/grafana/dashboards/
       volumes:
         - name: data
           persistentVolumeClaim:
@@ -58,3 +62,9 @@ spec:
         - name: datasources
           configMap:
             name: grafana-datasources
+        - name: dashboards-provider
+          configMap:
+            name: grafana-dashboards-provider
+        - name: dashboards
+          configMap:
+            name: grafana-dashboards

k8s/grafana/kustomization.yaml

@@ -7,9 +7,18 @@ resources:
   - service.yaml
   - ingress.yaml
 
-# Datasources only, as in the Ansible role. Dashboard provisioning is the
-# Phase 6 deliverable and belongs in this directory.
 configMapGenerator:
   - name: grafana-datasources
     files:
       - datasources.yml
+  - name: grafana-dashboards-provider
+    files:
+      - dashboards-provider.yml
+  - name: grafana-dashboards
+    files:
+      - dashboards/overview.json
+      - dashboards/red-fraud.json
+      - dashboards/red-gateway.json
+      - dashboards/red-ledger.json
+      - dashboards/red-sanctions.json
+      - dashboards/red-insights.json

k8s/prometheus/kustomization.yaml

@@ -14,3 +14,4 @@ configMapGenerator:
   - name: prometheus-rules
     files:
       - rules/host.rules.yml
+      - rules/ledgerline.rules.yml

k8s/prometheus/rules/ledgerline.rules.yml

@@ -0,0 +1,61 @@
+# Ledgerline alerting rules.
+#
+# ALERTING CONTRACT -- every alert in this directory (except scratch.*) must carry:
+#
+#   labels (machine-readable; Alertmanager routes and groups on these)
+#     severity   critical | warning | info
+#                critical = a human is needed now; warning = look today.
+#     team       who owns responding: platform | payments | risk | data
+#     service    which Ledgerline service, taken from the series' own label
+#                (aggregations must keep it: `by (service)`)
+#
+#   annotations (text for humans; never used for routing)
+#     summary      one line, no templating beyond the service name
+#     description  what is wrong, with the measured value and the threshold
+#     runbook_url  what to do about it (docs/runbooks/<kebab-alert-name>.md)
+#     dashboard    where to look first (the dashboard that explains the alert)
+#
+# observability/alerts/check_contract.py enforces this; run it before applying.
+#
+# `team` is static per alert rule, not per service: it says who answers this
+# *class* of alert. Per-service ownership would need a join against an
+# ownership metric or one rule per team.
+groups:
+  - name: ledgerline.rules
+    rules:
+      # All seven targets are scraped through one Traefik address, so a proxy
+      # failure takes every target down at once and fires seven alerts.
+      - alert: LedgerlineTargetDown
+        expr: up{job="ledgerline"} == 0
+        for: 2m
+        labels:
+          severity: critical
+          team: platform
+        annotations:
+          summary: "{{ $labels.service }} is not being scraped"
+          description: "Prometheus has failed to scrape {{ $labels.service }} for more than two minutes."
+          runbook_url: "https://registry.mohsen-hassani.com/mohsen_hassani/LedgerLine/src/branch/main/docs/runbooks/ledgerline-target-down.md"
+          dashboard: "https://grafana.vione.cloud/d/overview"
+
+      # 5xx only (a 404 is the client's mistake). The second condition keeps a
+      # service with almost no traffic from alerting on one failed request:
+      # below 0.05 req/s (about 15 requests per 5m) the ratio is noise. When
+      # there is no traffic at all the ratio is NaN and nothing fires.
+      - alert: LedgerlineHighErrorRatio
+        expr: |
+          (
+            sum by (service) (rate(http_server_request_duration_seconds_count{job="ledgerline",http_response_status_code=~"5.."}[5m]))
+            /
+            sum by (service) (rate(http_server_request_duration_seconds_count{job="ledgerline"}[5m]))
+          ) > 0.05
+          and
+          sum by (service) (rate(http_server_request_duration_seconds_count{job="ledgerline"}[5m])) > 0.05
+        for: 5m
+        labels:
+          severity: warning
+          team: payments
+        annotations:
+          summary: "{{ $labels.service }} is returning 5xx errors"
+          description: "{{ $labels.service }} has had a 5xx ratio of {{ $value | humanizePercentage }} for five minutes (threshold 5%)."
+          runbook_url: "https://registry.mohsen-hassani.com/mohsen_hassani/LedgerLine/src/branch/main/docs/runbooks/ledgerline-high-error-ratio.md"
+          dashboard: "https://grafana.vione.cloud/d/red-{{ $labels.service }}"

observability/alerts/check_contract.py

@@ -0,0 +1,58 @@
+"""Check every alert rule against the alerting contract (see the header of
+k8s/prometheus/rules/ledgerline.rules.yml).
+
+    python observability/alerts/check_contract.py
+
+Needs PyYAML. Skips scratch.* and host.* rule files.
+"""
+import sys
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(__file__).resolve().parents[2]
+RULES = ROOT / "k8s" / "prometheus" / "rules"
+
+SEVERITIES = {"critical", "warning", "info"}
+TEAMS = {"platform", "payments", "risk", "data"}  # placeholders: edit to taste
+ANNOTATIONS = ("summary", "description", "runbook_url", "dashboard")
+RUNBOOK_MARKER = "/docs/runbooks/"
+
+
+def check(path: Path) -> list[str]:
+    problems = []
+    for group in yaml.safe_load(path.read_text())["groups"]:
+        for rule in group["rules"]:
+            name = rule.get("alert")
+            if name is None:  # recording rule
+                continue
+            where = f"{path.name}:{name}"
+            labels, ann = rule.get("labels", {}), rule.get("annotations", {})
+            if labels.get("severity") not in SEVERITIES:
+                problems.append(f"{where}: severity must be one of {sorted(SEVERITIES)}")
+            if labels.get("team") not in TEAMS:
+                problems.append(f"{where}: team must be one of {sorted(TEAMS)}")
+            for key in ANNOTATIONS:
+                if not ann.get(key):
+                    problems.append(f"{where}: missing annotation {key}")
+            url = ann.get("runbook_url", "")
+            if RUNBOOK_MARKER in url:
+                local = ROOT / "docs" / "runbooks" / url.rsplit("/", 1)[-1]
+                if not local.exists():
+                    problems.append(f"{where}: runbook {local.relative_to(ROOT)} does not exist")
+            elif url:
+                problems.append(f"{where}: runbook_url should point under {RUNBOOK_MARKER}")
+    return problems
+
+
+def main() -> int:
+    files = [p for p in sorted(RULES.glob("*.yml")) if p.name.split(".")[0] not in ("scratch", "host")]
+    problems = [msg for p in files for msg in check(p)]
+    for msg in problems:
+        print(msg)
+    print(f"{len(files)} file(s) checked, {len(problems)} problem(s)")
+    return 1 if problems else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

observability/dashboards/generate.py

@@ -0,0 +1,213 @@
+"""Generate Grafana dashboard JSON into k8s/grafana/dashboards/.
+
+    python -m venv .venv && .venv/bin/pip install -r requirements.txt
+    .venv/bin/python generate.py
+"""
+import json
+from pathlib import Path
+
+from grafana_foundation_sdk.builders.dashboard import Dashboard
+from grafana_foundation_sdk.builders.prometheus import Dataquery as PromQuery
+from grafana_foundation_sdk.builders.nodegraph import Panel as NodeGraph
+from grafana_foundation_sdk.builders.tempo import TempoQuery
+from grafana_foundation_sdk.builders.timeseries import Panel as Timeseries
+from grafana_foundation_sdk.cog.encoder import JSONEncoder
+from grafana_foundation_sdk.models.dashboard import DataSourceRef
+
+OUT_DIR = Path(__file__).resolve().parents[2] / "k8s" / "grafana" / "dashboards"
+
+PROMETHEUS = DataSourceRef(type_val="prometheus", uid="prometheus")
+
+
+# Audit and Notifier are consumers and does not have HTTP, so commented them out for now
+SERVICES = [
+    "gateway",
+    "ledger",
+    "fraud",
+    "sanctions",
+    "insights",
+    # "audit",
+    # "notifier",
+]
+
+
+def prom(expr: str, ref_id: str, legend: str) -> PromQuery:
+    return (
+        PromQuery()
+        .expr(expr)
+        .ref_id(ref_id)
+        .legend_format(legend)
+        .datasource(PROMETHEUS)
+    )
+
+
+def red_dashboard(service: str) -> Dashboard:
+    """Rate, Errors, Duration for one service, in the order you'd ask them
+    during an incident: is it serving, is it failing, is it slow."""
+    # `service` and `instance` carry the same value on these series.
+    sel = f'job="ledgerline",service="{service}"'
+    count = f"http_server_request_duration_seconds_count{{{sel}}}"
+    bucket = f"http_server_request_duration_seconds_bucket{{{sel}}}"
+
+    def quantile(q: float) -> str:
+        return f"histogram_quantile({q}, sum by (le) (rate({bucket}[5m])))"
+
+    rate = (
+        Timeseries()
+        .title("Request rate")
+        .datasource(PROMETHEUS)
+        .unit("reqps")
+        .with_target(
+            prom(
+                f"sum by (http_route) (rate({count}[5m]))",
+                "rate",
+                "{{http_route}}",
+            )
+        )
+    )
+
+    errors = (
+        Timeseries()
+        .title("Error ratio")
+        .datasource(PROMETHEUS)
+        .unit("percentunit")
+        .min(0)
+        .with_target(
+            prom(
+                # 5xx only: a 404 is the client's mistake, not the service failing.
+                # `or vector(0)` keeps the line at 0 when no 5xx series exists yet;
+                # with no traffic the denominator is 0 and the result is NaN (a gap).
+                f'(sum(rate({count[:-1]},http_response_status_code=~"5.."}}[5m])) or vector(0))'
+                f" / sum(rate({count}[5m]))",
+                "errors",
+                "5xx ratio",
+            )
+        )
+    )
+
+    duration = (
+        Timeseries()
+        .title("Request duration")
+        .datasource(PROMETHEUS)
+        .unit("s")
+        # One query per quantile. Aggregate the histogram *buckets* first and
+        # compute the quantile after; never average per-instance quantiles.
+        .with_target(prom(quantile(0.50), "p50", "p50"))
+        .with_target(prom(quantile(0.95), "p95", "p95"))
+        .with_target(prom(quantile(0.99), "p99", "p99"))
+    )
+
+    return (
+        Dashboard(f"RED: {service}")
+        .uid(f"red-{service}")
+        .tags(["generated", "red"])
+        .time("now-1h", "now")
+        .refresh("15s")
+        .readonly()
+        .with_panel(rate)
+        .with_panel(errors)
+        .with_panel(duration)
+    )
+
+
+TEMPO = DataSourceRef(type_val="tempo", uid="tempo")
+
+
+def overview_dashboard() -> Dashboard:
+    """System-wide: who calls whom, is the async hop keeping up, is a pool
+    under pressure. Not per service, so no `service` variable."""
+
+    # Not a backend query: Grafana's browser builds the node graph from the
+    # Prometheus span metrics that Tempo's metrics-generator writes (wired by
+    # `serviceMap.datasourceUid` in datasources.yml). Calling it through
+    # /api/ds/query fails with "unsupported query type", so this panel can only
+    # be checked by looking at it in the browser.
+    service_graph = (
+        NodeGraph()
+        .title("Service graph")
+        .datasource(TEMPO)
+        .span(24)
+        .height(12)
+        .with_target(
+            TempoQuery()
+            .datasource(TEMPO)
+            .ref_id("graph")
+            .query_type("serviceMap")
+        )
+    )
+
+    lag = (
+        Timeseries()
+        .title("Stream consumer lag")
+        .datasource(PROMETHEUS)
+        .span(12)
+        .unit("none")
+        .with_target(
+            prom(
+                "ledgerline_stream_lag",
+                "lag",
+                "{{consumer_group}}",
+            )
+        )
+    )
+
+    pool = (
+        Timeseries()
+        .title("DB pool connections in use")
+        .datasource(PROMETHEUS)
+        .span(12)
+        .unit("none")
+        .with_target(
+            prom(
+                # "In use", not a percentage: the app exports no pool *size*.
+                # What would you need to turn this into saturation, and where
+                # could it come from? (Also look at the labels of
+                # `db_client_connections_usage_connections{state=...}`.)
+                "sum by (service, pool) (ledgerline_db_pool_in_use)",
+                "pool",
+                "{{service}} / {{pool}}",
+            )
+        )
+    )
+
+    return (
+        Dashboard("System overview")
+        .uid("overview")
+        .tags(["generated", "overview"])
+        .time("now-1h", "now")
+        .refresh("15s")
+        .readonly()
+        .with_panel(service_graph)
+        .with_panel(lag)
+        .with_panel(pool)
+    )
+
+
+DASHBOARDS = {
+    "overview": overview_dashboard,
+    **{f"red-{svc}": (lambda s=svc: red_dashboard(s)) for svc in SERVICES},
+}
+
+# A ConfigMap is capped at ~1 MiB, and client-side `kubectl apply` stores a
+# second copy in an annotation, so keep well under half of it.
+MAX_TOTAL_BYTES = 400_000
+
+
+def main() -> None:
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    rendered = {
+        name: json.dumps(fn().build(), cls=JSONEncoder, indent=2, sort_keys=True) + "\n"
+        for name, fn in DASHBOARDS.items()
+    }
+    for name, text in rendered.items():
+        assert "FIXME" not in text, f"{name}: unfilled FIXME in generated JSON"
+    total = sum(len(t) for t in rendered.values())
+    assert total < MAX_TOTAL_BYTES, f"{total} bytes of dashboards; ConfigMap limit"
+    for name, text in rendered.items():
+        (OUT_DIR / f"{name}.json").write_text(text)
+        print(f"wrote {OUT_DIR / f'{name}.json'}")
+    print(f"{total} bytes total")
+
+
+if __name__ == "__main__":
+    main()

observability/dashboards/requirements.txt

@@ -0,0 +1,3 @@
+# Epoch!grafana-version. The unpinned default is the Grafana 10.1 schema (highest
+# epoch = oldest schema); 11.6.0 is the newest published, loads fine in 12.1.1.
+grafana-foundation-sdk==1769699379!11.6.0