Add dashboards and alerts as code, mark Phase 6 done
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b50f8c3b24
commit
fef735c922
+53
-3
@@ -2,10 +2,60 @@
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
|
||||
|
||||
**Status:** Not started
|
||||
**Status:** Done
|
||||
|
||||
**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
|
||||
dashboarding.
|
||||
**Message:** Done 2026-10-25. Both Done-when criteria verified live: Grafana's
|
||||
resources and its volume were deleted and `kubectl apply -k k8s/` rebuilt all
|
||||
six dashboards and four datasources from the files (fresh PVC, every dashboard
|
||||
`provisioned: true`), and every dashboard's version history shows only the
|
||||
provisioner as author.
|
||||
|
||||
- **Build steps done:** 1, 2, 3, 4 (`tracesToMetrics` added; the
|
||||
`tracesToLogsV2` question was not answered) and 5.
|
||||
- **Dashboards:** `observability/dashboards/generate.py` (Foundation SDK)
|
||||
writes `k8s/grafana/dashboards/*.json`: `red-<service>` for the five HTTP
|
||||
services (gateway, ledger, fraud, sanctions, insights; audit and notifier
|
||||
serve no HTTP, so have none) and `overview` (service graph, stream lag, pool
|
||||
in use). No error-budget panel yet (Phase 8). The app exports no pool *size*,
|
||||
so the pool panel shows "in use", not saturation. The generator refuses to
|
||||
write while a `FIXME` remains and checks the total size against the
|
||||
ConfigMap limit.
|
||||
- **Alerts:** Prometheus rule files, not Grafana alerting, delivering to
|
||||
Alertmanager (`default` receiver is still a no-op). `LedgerlineTargetDown`
|
||||
and `LedgerlineHighErrorRatio` in `k8s/prometheus/rules/ledgerline.rules.yml`,
|
||||
under an alerting contract (labels `severity`/`team`/`service`; annotations
|
||||
`summary`/`description`/`runbook_url`/`dashboard`) enforced by
|
||||
`observability/alerts/check_contract.py`, with runbooks in `docs/runbooks/`.
|
||||
A scratch alert was seen in Alertmanager; `LedgerlineHighErrorRatio` was seen
|
||||
`pending` on real traffic, but its firing and rendered annotations were not
|
||||
verified. The runbook URLs are a guessed Gitea-style path and returned 404
|
||||
before the files were pushed.
|
||||
- **Not done:** answering the `tracesToLogsV2` / `filterByTraceID` question
|
||||
(the `QUESTION` comment is still in `k8s/grafana/datasources.yml`); clicking
|
||||
"Related metrics" on a span to confirm `tracesToMetrics` end to end; a real
|
||||
firing of either `ledgerline` alert; Grafana-managed alerting (Phase 8
|
||||
decides).
|
||||
- **Carry forward:**
|
||||
- The SDK's PyPI versions are `<epoch>!<grafana-version>`, and pip's default
|
||||
picks the *oldest* schema (10.1). Pin `grafana-foundation-sdk==1769699379!11.6.0`
|
||||
(newest published is the 11.6 schema; Grafana 12.1.1 loads it).
|
||||
- Provisioned dashboards can't be deleted from the UI or API in Grafana
|
||||
12.1.1 whatever `disableDeletion` says (the error is "provisioned dashboard
|
||||
cannot be deleted"). Removing the *file* removes the dashboard. The destroy
|
||||
test is `kubectl kustomize k8s/grafana | ssh falken kubectl delete -n
|
||||
ledgerline-observability -f -` then re-apply. **Do not run `kubectl delete
|
||||
-k k8s/`**: the root kustomization includes the Prometheus, Loki and Tempo
|
||||
PVCs and would delete all telemetry.
|
||||
- The configMap hash suffix rolls the pod on every change and leaves old
|
||||
hashed ConfigMaps behind (plain `apply` doesn't prune them).
|
||||
- Tempo's service map is built in the browser from Prometheus queries; the
|
||||
`serviceMap` query type returns "unsupported query type" through
|
||||
`/api/ds/query`, so that panel can only be checked by eye.
|
||||
- An unlabelled series renders as "Value" in a legend. The gateway's 404s have
|
||||
no `http_route`, so the rate panel shows one.
|
||||
- `team` on an alert is static per rule (who answers the class of alert), not
|
||||
per service. Phase 8's burn-rate rules belong in `k8s/prometheus/rules/`
|
||||
under the same contract.
|
||||
|
||||
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
|
||||
recreate in seconds.*
|
||||
|
||||
@@ -21,7 +21,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
| 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
||||
| 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
||||
| 5 | [`phase-5.md`](phase-5.md) | Done | Metrics | 2 evenings | RED per service, service map, exemplars |
|
||||
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
|
||||
| 6 | [`phase-6.md`](phase-6.md) | Done | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
|
||||
| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
|
||||
| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
|
||||
| 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
|
||||
@@ -56,7 +56,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
`phase-5.md` (second metrics path and the TraceQL test migration not done).
|
||||
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
||||
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
|
||||
"destroy and recreate in seconds" is a real, repeatable command.
|
||||
"destroy and recreate in seconds" is a real, repeatable command. **Done** —
|
||||
see `phase-6.md` (alerts as Prometheus rules under a contract; the
|
||||
`tracesToLogsV2` question and a real alert firing not done).
|
||||
- **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
|
||||
be averaged, what coordinated omission is, and drive Ledgerline with
|
||||
arrival-rate executors to get a clean before/after load profile.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# LedgerlineHighErrorRatio
|
||||
|
||||
**Fires when:** a service's 5xx ratio is above 5% for 5 minutes, with more than
|
||||
0.05 requests/s (so it is not one failed request on an idle service).
|
||||
**Severity:** warning · **Team:** payments
|
||||
|
||||
## What it means
|
||||
|
||||
A meaningful share of the requests this service serves are failing with a server
|
||||
error. 4xx responses (for example a 404) are deliberately not counted.
|
||||
|
||||
## First five minutes
|
||||
|
||||
1. **Look at the `RED: <service>` dashboard** (linked in the alert). Is the error
|
||||
ratio still elevated, or already back to normal? Did the request rate or the
|
||||
latency change at the same time?
|
||||
2. **Find a failing request.** In Tempo, search for the service with an error
|
||||
status, or in Loki: `{service_name="<service>"} | json | level="ERROR"`.
|
||||
3. **Follow one trace.** The `trace_id` in the log line opens the whole request
|
||||
across services. Look for the first span in the trace with an error status.
|
||||
4. **Is it only this service?** The System overview's service graph shows whether
|
||||
its callers or callees are failing as well.
|
||||
|
||||
## Escalate when
|
||||
|
||||
The ratio is still rising after ten minutes, or the failing requests are on the
|
||||
critical path (transactions), not a side route.
|
||||
@@ -0,0 +1,29 @@
|
||||
# LedgerlineTargetDown
|
||||
|
||||
**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes.
|
||||
**Severity:** critical · **Team:** platform
|
||||
|
||||
## What it means
|
||||
|
||||
Prometheus scrapes all seven services through one address,
|
||||
`https://ledgerline.vione.cloud/metrics/<service>` (czy's Traefik, basic auth).
|
||||
"Down" means that scrape failed. It does **not** by itself prove the service is
|
||||
down: the service, the proxy, or the credentials can each cause it.
|
||||
|
||||
## First five minutes
|
||||
|
||||
1. **How many targets?** Open the alert in Alertmanager. One service alerting is
|
||||
likely that service. All seven at once points at the shared path (Traefik on
|
||||
czy, DNS, the scrape credentials), not at the services.
|
||||
2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and
|
||||
read its `Last error`.
|
||||
3. **Is the container running?** `ssh czy docker compose ps` for the service.
|
||||
4. **What did it last say?** `ssh czy docker logs ledgerline-<service>-1`, or in
|
||||
Loki: `{service_name="<service>"}`.
|
||||
5. **Is traffic still flowing?** Open the `RED: <service>` dashboard. Traces and
|
||||
logs can still be arriving while only the metrics scrape is broken.
|
||||
|
||||
## Escalate when
|
||||
|
||||
The shared path is down (all seven targets), or the service is down and not
|
||||
recovering after a restart.
|
||||
Reference in New Issue
Block a user