diff --git a/docs/roadmap/phase-6.md b/docs/roadmap/phase-6.md index a6fd270..0e87712 100644 --- a/docs/roadmap/phase-6.md +++ b/docs/roadmap/phase-6.md @@ -2,10 +2,60 @@ *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)* -**Status:** Not started +**Status:** Done -**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth -dashboarding. +**Message:** Done 2026-10-25. Both Done-when criteria verified live: Grafana's +resources and its volume were deleted and `kubectl apply -k k8s/` rebuilt all +six dashboards and four datasources from the files (fresh PVC, every dashboard +`provisioned: true`), and every dashboard's version history shows only the +provisioner as author. + +- **Build steps done:** 1, 2, 3, 4 (`tracesToMetrics` added; the + `tracesToLogsV2` question was not answered) and 5. +- **Dashboards:** `observability/dashboards/generate.py` (Foundation SDK) + writes `k8s/grafana/dashboards/*.json`: `red-` for the five HTTP + services (gateway, ledger, fraud, sanctions, insights; audit and notifier + serve no HTTP, so have none) and `overview` (service graph, stream lag, pool + in use). No error-budget panel yet (Phase 8). The app exports no pool *size*, + so the pool panel shows "in use", not saturation. The generator refuses to + write while a `FIXME` remains and checks the total size against the + ConfigMap limit. +- **Alerts:** Prometheus rule files, not Grafana alerting, delivering to + Alertmanager (`default` receiver is still a no-op). `LedgerlineTargetDown` + and `LedgerlineHighErrorRatio` in `k8s/prometheus/rules/ledgerline.rules.yml`, + under an alerting contract (labels `severity`/`team`/`service`; annotations + `summary`/`description`/`runbook_url`/`dashboard`) enforced by + `observability/alerts/check_contract.py`, with runbooks in `docs/runbooks/`. + A scratch alert was seen in Alertmanager; `LedgerlineHighErrorRatio` was seen + `pending` on real traffic, but its firing and rendered annotations were not + verified. The runbook URLs are a guessed Gitea-style path and returned 404 + before the files were pushed. +- **Not done:** answering the `tracesToLogsV2` / `filterByTraceID` question + (the `QUESTION` comment is still in `k8s/grafana/datasources.yml`); clicking + "Related metrics" on a span to confirm `tracesToMetrics` end to end; a real + firing of either `ledgerline` alert; Grafana-managed alerting (Phase 8 + decides). +- **Carry forward:** + - The SDK's PyPI versions are `!`, and pip's default + picks the *oldest* schema (10.1). Pin `grafana-foundation-sdk==1769699379!11.6.0` + (newest published is the 11.6 schema; Grafana 12.1.1 loads it). + - Provisioned dashboards can't be deleted from the UI or API in Grafana + 12.1.1 whatever `disableDeletion` says (the error is "provisioned dashboard + cannot be deleted"). Removing the *file* removes the dashboard. The destroy + test is `kubectl kustomize k8s/grafana | ssh falken kubectl delete -n + ledgerline-observability -f -` then re-apply. **Do not run `kubectl delete + -k k8s/`**: the root kustomization includes the Prometheus, Loki and Tempo + PVCs and would delete all telemetry. + - The configMap hash suffix rolls the pod on every change and leaves old + hashed ConfigMaps behind (plain `apply` doesn't prune them). + - Tempo's service map is built in the browser from Prometheus queries; the + `serviceMap` query type returns "unsupported query type" through + `/api/ds/query`, so that panel can only be checked by eye. + - An unlabelled series renders as "Value" in a legend. The gateway's 404s have + no `http_route`, so the rate panel shows one. + - `team` on an alert is static per rule (who answers the class of alert), not + per service. Phase 8's burn-rate rules belong in `k8s/prometheus/rules/` + under the same contract. **Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and recreate in seconds.* diff --git a/docs/roadmap/summary.md b/docs/roadmap/summary.md index 1a84bf4..ee9208b 100644 --- a/docs/roadmap/summary.md +++ b/docs/roadmap/summary.md @@ -21,7 +21,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. | 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | | 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation | | 5 | [`phase-5.md`](phase-5.md) | Done | Metrics | 2 evenings | RED per service, service map, exemplars | -| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards | +| 6 | [`phase-6.md`](phase-6.md) | Done | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards | | 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles | | 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts | | 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` | @@ -56,7 +56,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. `phase-5.md` (second metrics path and the TraceQL test migration not done). - **6 — Dashboards and alerts as code.** Generate dashboard JSON with the Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so - "destroy and recreate in seconds" is a real, repeatable command. + "destroy and recreate in seconds" is a real, repeatable command. **Done** — + see `phase-6.md` (alerts as Prometheus rules under a contract; the + `tracesToLogsV2` question and a real alert firing not done). - **7 — Load testing with k6, and percentiles.** Learn why percentiles can't be averaged, what coordinated omission is, and drive Ledgerline with arrival-rate executors to get a clean before/after load profile. diff --git a/docs/runbooks/ledgerline-high-error-ratio.md b/docs/runbooks/ledgerline-high-error-ratio.md new file mode 100644 index 0000000..16fb12e --- /dev/null +++ b/docs/runbooks/ledgerline-high-error-ratio.md @@ -0,0 +1,27 @@ +# LedgerlineHighErrorRatio + +**Fires when:** a service's 5xx ratio is above 5% for 5 minutes, with more than +0.05 requests/s (so it is not one failed request on an idle service). +**Severity:** warning · **Team:** payments + +## What it means + +A meaningful share of the requests this service serves are failing with a server +error. 4xx responses (for example a 404) are deliberately not counted. + +## First five minutes + +1. **Look at the `RED: ` dashboard** (linked in the alert). Is the error + ratio still elevated, or already back to normal? Did the request rate or the + latency change at the same time? +2. **Find a failing request.** In Tempo, search for the service with an error + status, or in Loki: `{service_name=""} | json | level="ERROR"`. +3. **Follow one trace.** The `trace_id` in the log line opens the whole request + across services. Look for the first span in the trace with an error status. +4. **Is it only this service?** The System overview's service graph shows whether + its callers or callees are failing as well. + +## Escalate when + +The ratio is still rising after ten minutes, or the failing requests are on the +critical path (transactions), not a side route. diff --git a/docs/runbooks/ledgerline-target-down.md b/docs/runbooks/ledgerline-target-down.md new file mode 100644 index 0000000..acfeacf --- /dev/null +++ b/docs/runbooks/ledgerline-target-down.md @@ -0,0 +1,29 @@ +# LedgerlineTargetDown + +**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes. +**Severity:** critical · **Team:** platform + +## What it means + +Prometheus scrapes all seven services through one address, +`https://ledgerline.vione.cloud/metrics/` (czy's Traefik, basic auth). +"Down" means that scrape failed. It does **not** by itself prove the service is +down: the service, the proxy, or the credentials can each cause it. + +## First five minutes + +1. **How many targets?** Open the alert in Alertmanager. One service alerting is + likely that service. All seven at once points at the shared path (Traefik on + czy, DNS, the scrape credentials), not at the services. +2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and + read its `Last error`. +3. **Is the container running?** `ssh czy docker compose ps` for the service. +4. **What did it last say?** `ssh czy docker logs ledgerline--1`, or in + Loki: `{service_name=""}`. +5. **Is traffic still flowing?** Open the `RED: ` dashboard. Traces and + logs can still be arriving while only the metrics scrape is broken. + +## Escalate when + +The shared path is down (all seven targets), or the service is down and not +recovering after a restart. diff --git a/k8s/grafana/dashboards-provider.yml b/k8s/grafana/dashboards-provider.yml new file mode 100644 index 0000000..2e3273f --- /dev/null +++ b/k8s/grafana/dashboards-provider.yml @@ -0,0 +1,13 @@ +apiVersion: 1 + +providers: + - name: ledgerline + orgId: 1 + type: file + folder: Ledgerline + options: + path: /etc/grafana/dashboards/ + allowUiUpdates: false + disableDeletion: false + updateIntervalSeconds: 15 + foldersFromFilesStructure: false diff --git a/k8s/grafana/dashboards/overview.json b/k8s/grafana/dashboards/overview.json new file mode 100644 index 0000000..1cfbaa4 --- /dev/null +++ b/k8s/grafana/dashboards/overview.json @@ -0,0 +1,115 @@ +{ + "annotations": {}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "panels": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 12, + "w": 24, + "x": 0, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "filters": [], + "queryType": "serviceMap", + "refId": "graph" + } + ], + "title": "Service graph", + "transparent": false, + "type": "nodeGraph" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 12 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "ledgerline_stream_lag", + "legendFormat": "{{consumer_group}}", + "refId": "lag" + } + ], + "title": "Stream consumer lag", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 12 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by (service, pool) (ledgerline_db_pool_in_use)", + "legendFormat": "{{service}} / {{pool}}", + "refId": "pool" + } + ], + "title": "DB pool connections in use", + "transparent": false, + "type": "timeseries" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": [ + "generated", + "overview" + ], + "templating": {}, + "time": { + "from": "now-1h", + "to": "now" + }, + "timezone": "browser", + "title": "System overview", + "uid": "overview" +} diff --git a/k8s/grafana/dashboards/red-fraud.json b/k8s/grafana/dashboards/red-fraud.json new file mode 100644 index 0000000..968f38b --- /dev/null +++ b/k8s/grafana/dashboards/red-fraud.json @@ -0,0 +1,140 @@ +{ + "annotations": {}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"fraud\"}[5m]))", + "legendFormat": "{{http_route}}", + "refId": "rate" + } + ], + "title": "Request rate", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "min": 0, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"fraud\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"fraud\"}[5m]))", + "legendFormat": "5xx ratio", + "refId": "errors" + } + ], + "title": "Error ratio", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 9 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"fraud\"}[5m])))", + "legendFormat": "p50", + "refId": "p50" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"fraud\"}[5m])))", + "legendFormat": "p95", + "refId": "p95" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"fraud\"}[5m])))", + "legendFormat": "p99", + "refId": "p99" + } + ], + "title": "Request duration", + "transparent": false, + "type": "timeseries" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": [ + "generated", + "red" + ], + "templating": {}, + "time": { + "from": "now-1h", + "to": "now" + }, + "timezone": "browser", + "title": "RED: fraud", + "uid": "red-fraud" +} diff --git a/k8s/grafana/dashboards/red-gateway.json b/k8s/grafana/dashboards/red-gateway.json new file mode 100644 index 0000000..2460036 --- /dev/null +++ b/k8s/grafana/dashboards/red-gateway.json @@ -0,0 +1,140 @@ +{ + "annotations": {}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"gateway\"}[5m]))", + "legendFormat": "{{http_route}}", + "refId": "rate" + } + ], + "title": "Request rate", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "min": 0, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"gateway\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"gateway\"}[5m]))", + "legendFormat": "5xx ratio", + "refId": "errors" + } + ], + "title": "Error ratio", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 9 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"gateway\"}[5m])))", + "legendFormat": "p50", + "refId": "p50" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"gateway\"}[5m])))", + "legendFormat": "p95", + "refId": "p95" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"gateway\"}[5m])))", + "legendFormat": "p99", + "refId": "p99" + } + ], + "title": "Request duration", + "transparent": false, + "type": "timeseries" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": [ + "generated", + "red" + ], + "templating": {}, + "time": { + "from": "now-1h", + "to": "now" + }, + "timezone": "browser", + "title": "RED: gateway", + "uid": "red-gateway" +} diff --git a/k8s/grafana/dashboards/red-insights.json b/k8s/grafana/dashboards/red-insights.json new file mode 100644 index 0000000..8b2a1bb --- /dev/null +++ b/k8s/grafana/dashboards/red-insights.json @@ -0,0 +1,140 @@ +{ + "annotations": {}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"insights\"}[5m]))", + "legendFormat": "{{http_route}}", + "refId": "rate" + } + ], + "title": "Request rate", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "min": 0, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"insights\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"insights\"}[5m]))", + "legendFormat": "5xx ratio", + "refId": "errors" + } + ], + "title": "Error ratio", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 9 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"insights\"}[5m])))", + "legendFormat": "p50", + "refId": "p50" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"insights\"}[5m])))", + "legendFormat": "p95", + "refId": "p95" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"insights\"}[5m])))", + "legendFormat": "p99", + "refId": "p99" + } + ], + "title": "Request duration", + "transparent": false, + "type": "timeseries" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": [ + "generated", + "red" + ], + "templating": {}, + "time": { + "from": "now-1h", + "to": "now" + }, + "timezone": "browser", + "title": "RED: insights", + "uid": "red-insights" +} diff --git a/k8s/grafana/dashboards/red-ledger.json b/k8s/grafana/dashboards/red-ledger.json new file mode 100644 index 0000000..0bbe16d --- /dev/null +++ b/k8s/grafana/dashboards/red-ledger.json @@ -0,0 +1,140 @@ +{ + "annotations": {}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"ledger\"}[5m]))", + "legendFormat": "{{http_route}}", + "refId": "rate" + } + ], + "title": "Request rate", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "min": 0, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"ledger\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"ledger\"}[5m]))", + "legendFormat": "5xx ratio", + "refId": "errors" + } + ], + "title": "Error ratio", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 9 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"ledger\"}[5m])))", + "legendFormat": "p50", + "refId": "p50" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"ledger\"}[5m])))", + "legendFormat": "p95", + "refId": "p95" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"ledger\"}[5m])))", + "legendFormat": "p99", + "refId": "p99" + } + ], + "title": "Request duration", + "transparent": false, + "type": "timeseries" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": [ + "generated", + "red" + ], + "templating": {}, + "time": { + "from": "now-1h", + "to": "now" + }, + "timezone": "browser", + "title": "RED: ledger", + "uid": "red-ledger" +} diff --git a/k8s/grafana/dashboards/red-sanctions.json b/k8s/grafana/dashboards/red-sanctions.json new file mode 100644 index 0000000..66e75ea --- /dev/null +++ b/k8s/grafana/dashboards/red-sanctions.json @@ -0,0 +1,140 @@ +{ + "annotations": {}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"sanctions\"}[5m]))", + "legendFormat": "{{http_route}}", + "refId": "rate" + } + ], + "title": "Request rate", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "min": 0, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 0 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"sanctions\",http_response_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_request_duration_seconds_count{job=\"ledgerline\",service=\"sanctions\"}[5m]))", + "legendFormat": "5xx ratio", + "refId": "errors" + } + ], + "title": "Error ratio", + "transparent": false, + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 9 + }, + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"sanctions\"}[5m])))", + "legendFormat": "p50", + "refId": "p50" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"sanctions\"}[5m])))", + "legendFormat": "p95", + "refId": "p95" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{job=\"ledgerline\",service=\"sanctions\"}[5m])))", + "legendFormat": "p99", + "refId": "p99" + } + ], + "title": "Request duration", + "transparent": false, + "type": "timeseries" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": [ + "generated", + "red" + ], + "templating": {}, + "time": { + "from": "now-1h", + "to": "now" + }, + "timezone": "browser", + "title": "RED: sanctions", + "uid": "red-sanctions" +} diff --git a/k8s/grafana/datasources.yml b/k8s/grafana/datasources.yml index 7d40066..f236a0a 100644 --- a/k8s/grafana/datasources.yml +++ b/k8s/grafana/datasources.yml @@ -38,6 +38,22 @@ datasources: filterByTraceID: true spanStartTimeShift: -1m spanEndTimeShift: 1m + # QUESTION (answer in your own words; verify by reading the LogQL that + # "Logs for this span" generates): why is `filterByTraceID: true` enough + # with no `tags` mapping here, and when would it not be? + tracesToMetrics: + datasourceUid: prometheus + # Wider than the span: rate() needs a few scrape intervals of data. + spanStartTimeShift: -2m + spanEndTimeShift: 2m + # span attribute -> Prometheus label on traces_spanmetrics_* + tags: + - key: service.name + value: service + queries: + # `$$` is the provisioning-file escape for a literal `$`. + - name: Request rate + query: sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m])) serviceMap: datasourceUid: prometheus nodeGraph: diff --git a/k8s/grafana/deployment.yaml b/k8s/grafana/deployment.yaml index 7b39db6..e672a52 100644 --- a/k8s/grafana/deployment.yaml +++ b/k8s/grafana/deployment.yaml @@ -51,6 +51,10 @@ spec: mountPath: /var/lib/grafana - name: datasources mountPath: /etc/grafana/provisioning/datasources + - name: dashboards-provider + mountPath: /etc/grafana/provisioning/dashboards + - name: dashboards + mountPath: /etc/grafana/dashboards/ volumes: - name: data persistentVolumeClaim: @@ -58,3 +62,9 @@ spec: - name: datasources configMap: name: grafana-datasources + - name: dashboards-provider + configMap: + name: grafana-dashboards-provider + - name: dashboards + configMap: + name: grafana-dashboards diff --git a/k8s/grafana/kustomization.yaml b/k8s/grafana/kustomization.yaml index 3db3407..ece0d31 100644 --- a/k8s/grafana/kustomization.yaml +++ b/k8s/grafana/kustomization.yaml @@ -7,9 +7,18 @@ resources: - service.yaml - ingress.yaml -# Datasources only, as in the Ansible role. Dashboard provisioning is the -# Phase 6 deliverable and belongs in this directory. configMapGenerator: - name: grafana-datasources files: - datasources.yml + - name: grafana-dashboards-provider + files: + - dashboards-provider.yml + - name: grafana-dashboards + files: + - dashboards/overview.json + - dashboards/red-fraud.json + - dashboards/red-gateway.json + - dashboards/red-ledger.json + - dashboards/red-sanctions.json + - dashboards/red-insights.json diff --git a/k8s/prometheus/kustomization.yaml b/k8s/prometheus/kustomization.yaml index ce8c09d..6043920 100644 --- a/k8s/prometheus/kustomization.yaml +++ b/k8s/prometheus/kustomization.yaml @@ -14,3 +14,4 @@ configMapGenerator: - name: prometheus-rules files: - rules/host.rules.yml + - rules/ledgerline.rules.yml diff --git a/k8s/prometheus/rules/ledgerline.rules.yml b/k8s/prometheus/rules/ledgerline.rules.yml new file mode 100644 index 0000000..0425ed9 --- /dev/null +++ b/k8s/prometheus/rules/ledgerline.rules.yml @@ -0,0 +1,61 @@ +# Ledgerline alerting rules. +# +# ALERTING CONTRACT -- every alert in this directory (except scratch.*) must carry: +# +# labels (machine-readable; Alertmanager routes and groups on these) +# severity critical | warning | info +# critical = a human is needed now; warning = look today. +# team who owns responding: platform | payments | risk | data +# service which Ledgerline service, taken from the series' own label +# (aggregations must keep it: `by (service)`) +# +# annotations (text for humans; never used for routing) +# summary one line, no templating beyond the service name +# description what is wrong, with the measured value and the threshold +# runbook_url what to do about it (docs/runbooks/.md) +# dashboard where to look first (the dashboard that explains the alert) +# +# observability/alerts/check_contract.py enforces this; run it before applying. +# +# `team` is static per alert rule, not per service: it says who answers this +# *class* of alert. Per-service ownership would need a join against an +# ownership metric or one rule per team. +groups: + - name: ledgerline.rules + rules: + # All seven targets are scraped through one Traefik address, so a proxy + # failure takes every target down at once and fires seven alerts. + - alert: LedgerlineTargetDown + expr: up{job="ledgerline"} == 0 + for: 2m + labels: + severity: critical + team: platform + annotations: + summary: "{{ $labels.service }} is not being scraped" + description: "Prometheus has failed to scrape {{ $labels.service }} for more than two minutes." + runbook_url: "https://registry.mohsen-hassani.com/mohsen_hassani/LedgerLine/src/branch/main/docs/runbooks/ledgerline-target-down.md" + dashboard: "https://grafana.vione.cloud/d/overview" + + # 5xx only (a 404 is the client's mistake). The second condition keeps a + # service with almost no traffic from alerting on one failed request: + # below 0.05 req/s (about 15 requests per 5m) the ratio is noise. When + # there is no traffic at all the ratio is NaN and nothing fires. + - alert: LedgerlineHighErrorRatio + expr: | + ( + sum by (service) (rate(http_server_request_duration_seconds_count{job="ledgerline",http_response_status_code=~"5.."}[5m])) + / + sum by (service) (rate(http_server_request_duration_seconds_count{job="ledgerline"}[5m])) + ) > 0.05 + and + sum by (service) (rate(http_server_request_duration_seconds_count{job="ledgerline"}[5m])) > 0.05 + for: 5m + labels: + severity: warning + team: payments + annotations: + summary: "{{ $labels.service }} is returning 5xx errors" + description: "{{ $labels.service }} has had a 5xx ratio of {{ $value | humanizePercentage }} for five minutes (threshold 5%)." + runbook_url: "https://registry.mohsen-hassani.com/mohsen_hassani/LedgerLine/src/branch/main/docs/runbooks/ledgerline-high-error-ratio.md" + dashboard: "https://grafana.vione.cloud/d/red-{{ $labels.service }}" diff --git a/observability/alerts/check_contract.py b/observability/alerts/check_contract.py new file mode 100644 index 0000000..c3afde5 --- /dev/null +++ b/observability/alerts/check_contract.py @@ -0,0 +1,58 @@ +"""Check every alert rule against the alerting contract (see the header of +k8s/prometheus/rules/ledgerline.rules.yml). + + python observability/alerts/check_contract.py + +Needs PyYAML. Skips scratch.* and host.* rule files. +""" +import sys +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[2] +RULES = ROOT / "k8s" / "prometheus" / "rules" + +SEVERITIES = {"critical", "warning", "info"} +TEAMS = {"platform", "payments", "risk", "data"} # placeholders: edit to taste +ANNOTATIONS = ("summary", "description", "runbook_url", "dashboard") +RUNBOOK_MARKER = "/docs/runbooks/" + + +def check(path: Path) -> list[str]: + problems = [] + for group in yaml.safe_load(path.read_text())["groups"]: + for rule in group["rules"]: + name = rule.get("alert") + if name is None: # recording rule + continue + where = f"{path.name}:{name}" + labels, ann = rule.get("labels", {}), rule.get("annotations", {}) + if labels.get("severity") not in SEVERITIES: + problems.append(f"{where}: severity must be one of {sorted(SEVERITIES)}") + if labels.get("team") not in TEAMS: + problems.append(f"{where}: team must be one of {sorted(TEAMS)}") + for key in ANNOTATIONS: + if not ann.get(key): + problems.append(f"{where}: missing annotation {key}") + url = ann.get("runbook_url", "") + if RUNBOOK_MARKER in url: + local = ROOT / "docs" / "runbooks" / url.rsplit("/", 1)[-1] + if not local.exists(): + problems.append(f"{where}: runbook {local.relative_to(ROOT)} does not exist") + elif url: + problems.append(f"{where}: runbook_url should point under {RUNBOOK_MARKER}") + return problems + + +def main() -> int: + files = [p for p in sorted(RULES.glob("*.yml")) if p.name.split(".")[0] not in ("scratch", "host")] + problems = [msg for p in files for msg in check(p)] + for msg in problems: + print(msg) + print(f"{len(files)} file(s) checked, {len(problems)} problem(s)") + return 1 if problems else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/observability/dashboards/generate.py b/observability/dashboards/generate.py new file mode 100644 index 0000000..3acbb97 --- /dev/null +++ b/observability/dashboards/generate.py @@ -0,0 +1,213 @@ +"""Generate Grafana dashboard JSON into k8s/grafana/dashboards/. + + python -m venv .venv && .venv/bin/pip install -r requirements.txt + .venv/bin/python generate.py +""" +import json +from pathlib import Path + +from grafana_foundation_sdk.builders.dashboard import Dashboard +from grafana_foundation_sdk.builders.prometheus import Dataquery as PromQuery +from grafana_foundation_sdk.builders.nodegraph import Panel as NodeGraph +from grafana_foundation_sdk.builders.tempo import TempoQuery +from grafana_foundation_sdk.builders.timeseries import Panel as Timeseries +from grafana_foundation_sdk.cog.encoder import JSONEncoder +from grafana_foundation_sdk.models.dashboard import DataSourceRef + +OUT_DIR = Path(__file__).resolve().parents[2] / "k8s" / "grafana" / "dashboards" + +PROMETHEUS = DataSourceRef(type_val="prometheus", uid="prometheus") + + +# Audit and Notifier are consumers and does not have HTTP, so commented them out for now +SERVICES = [ + "gateway", + "ledger", + "fraud", + "sanctions", + "insights", + # "audit", + # "notifier", +] + + +def prom(expr: str, ref_id: str, legend: str) -> PromQuery: + return ( + PromQuery() + .expr(expr) + .ref_id(ref_id) + .legend_format(legend) + .datasource(PROMETHEUS) + ) + + +def red_dashboard(service: str) -> Dashboard: + """Rate, Errors, Duration for one service, in the order you'd ask them + during an incident: is it serving, is it failing, is it slow.""" + # `service` and `instance` carry the same value on these series. + sel = f'job="ledgerline",service="{service}"' + count = f"http_server_request_duration_seconds_count{{{sel}}}" + bucket = f"http_server_request_duration_seconds_bucket{{{sel}}}" + + def quantile(q: float) -> str: + return f"histogram_quantile({q}, sum by (le) (rate({bucket}[5m])))" + + rate = ( + Timeseries() + .title("Request rate") + .datasource(PROMETHEUS) + .unit("reqps") + .with_target( + prom( + f"sum by (http_route) (rate({count}[5m]))", + "rate", + "{{http_route}}", + ) + ) + ) + + errors = ( + Timeseries() + .title("Error ratio") + .datasource(PROMETHEUS) + .unit("percentunit") + .min(0) + .with_target( + prom( + # 5xx only: a 404 is the client's mistake, not the service failing. + # `or vector(0)` keeps the line at 0 when no 5xx series exists yet; + # with no traffic the denominator is 0 and the result is NaN (a gap). + f'(sum(rate({count[:-1]},http_response_status_code=~"5.."}}[5m])) or vector(0))' + f" / sum(rate({count}[5m]))", + "errors", + "5xx ratio", + ) + ) + ) + + duration = ( + Timeseries() + .title("Request duration") + .datasource(PROMETHEUS) + .unit("s") + # One query per quantile. Aggregate the histogram *buckets* first and + # compute the quantile after; never average per-instance quantiles. + .with_target(prom(quantile(0.50), "p50", "p50")) + .with_target(prom(quantile(0.95), "p95", "p95")) + .with_target(prom(quantile(0.99), "p99", "p99")) + ) + + return ( + Dashboard(f"RED: {service}") + .uid(f"red-{service}") + .tags(["generated", "red"]) + .time("now-1h", "now") + .refresh("15s") + .readonly() + .with_panel(rate) + .with_panel(errors) + .with_panel(duration) + ) + + +TEMPO = DataSourceRef(type_val="tempo", uid="tempo") + + +def overview_dashboard() -> Dashboard: + """System-wide: who calls whom, is the async hop keeping up, is a pool + under pressure. Not per service, so no `service` variable.""" + + # Not a backend query: Grafana's browser builds the node graph from the + # Prometheus span metrics that Tempo's metrics-generator writes (wired by + # `serviceMap.datasourceUid` in datasources.yml). Calling it through + # /api/ds/query fails with "unsupported query type", so this panel can only + # be checked by looking at it in the browser. + service_graph = ( + NodeGraph() + .title("Service graph") + .datasource(TEMPO) + .span(24) + .height(12) + .with_target( + TempoQuery() + .datasource(TEMPO) + .ref_id("graph") + .query_type("serviceMap") + ) + ) + + lag = ( + Timeseries() + .title("Stream consumer lag") + .datasource(PROMETHEUS) + .span(12) + .unit("none") + .with_target( + prom( + "ledgerline_stream_lag", + "lag", + "{{consumer_group}}", + ) + ) + ) + + pool = ( + Timeseries() + .title("DB pool connections in use") + .datasource(PROMETHEUS) + .span(12) + .unit("none") + .with_target( + prom( + # "In use", not a percentage: the app exports no pool *size*. + # What would you need to turn this into saturation, and where + # could it come from? (Also look at the labels of + # `db_client_connections_usage_connections{state=...}`.) + "sum by (service, pool) (ledgerline_db_pool_in_use)", + "pool", + "{{service}} / {{pool}}", + ) + ) + ) + + return ( + Dashboard("System overview") + .uid("overview") + .tags(["generated", "overview"]) + .time("now-1h", "now") + .refresh("15s") + .readonly() + .with_panel(service_graph) + .with_panel(lag) + .with_panel(pool) + ) + + +DASHBOARDS = { + "overview": overview_dashboard, + **{f"red-{svc}": (lambda s=svc: red_dashboard(s)) for svc in SERVICES}, +} + +# A ConfigMap is capped at ~1 MiB, and client-side `kubectl apply` stores a +# second copy in an annotation, so keep well under half of it. +MAX_TOTAL_BYTES = 400_000 + + +def main() -> None: + OUT_DIR.mkdir(parents=True, exist_ok=True) + rendered = { + name: json.dumps(fn().build(), cls=JSONEncoder, indent=2, sort_keys=True) + "\n" + for name, fn in DASHBOARDS.items() + } + for name, text in rendered.items(): + assert "FIXME" not in text, f"{name}: unfilled FIXME in generated JSON" + total = sum(len(t) for t in rendered.values()) + assert total < MAX_TOTAL_BYTES, f"{total} bytes of dashboards; ConfigMap limit" + for name, text in rendered.items(): + (OUT_DIR / f"{name}.json").write_text(text) + print(f"wrote {OUT_DIR / f'{name}.json'}") + print(f"{total} bytes total") + + +if __name__ == "__main__": + main() diff --git a/observability/dashboards/requirements.txt b/observability/dashboards/requirements.txt new file mode 100644 index 0000000..6239d24 --- /dev/null +++ b/observability/dashboards/requirements.txt @@ -0,0 +1,3 @@ +# Epoch!grafana-version. The unpinned default is the Grafana 10.1 schema (highest +# epoch = oldest schema); 11.6.0 is the newest published, loads fine in 12.1.1. +grafana-foundation-sdk==1769699379!11.6.0