Add dashboards and alerts as code, mark Phase 6 done

Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.

Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-25 20:01:52 +03:30
co-authored by Claude Sonnet 5.5
parent b50f8c3b24
commit fef735c922
19 changed files with 1314 additions and 7 deletions
+29
View File
@@ -0,0 +1,29 @@
# LedgerlineTargetDown
**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes.
**Severity:** critical · **Team:** platform
## What it means
Prometheus scrapes all seven services through one address,
`https://ledgerline.vione.cloud/metrics/<service>` (czy's Traefik, basic auth).
"Down" means that scrape failed. It does **not** by itself prove the service is
down: the service, the proxy, or the credentials can each cause it.
## First five minutes
1. **How many targets?** Open the alert in Alertmanager. One service alerting is
likely that service. All seven at once points at the shared path (Traefik on
czy, DNS, the scrape credentials), not at the services.
2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and
read its `Last error`.
3. **Is the container running?** `ssh czy docker compose ps` for the service.
4. **What did it last say?** `ssh czy docker logs ledgerline-<service>-1`, or in
Loki: `{service_name="<service>"}`.
5. **Is traffic still flowing?** Open the `RED: <service>` dashboard. Traces and
logs can still be arriving while only the metrics scrape is broken.
## Escalate when
The shared path is down (all seven targets), or the service is down and not
recovering after a restart.