Add dashboards and alerts as code, mark Phase 6 done
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5.5
parent
b50f8c3b24
commit
fef735c922
@@ -0,0 +1,29 @@
|
||||
# LedgerlineTargetDown
|
||||
|
||||
**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes.
|
||||
**Severity:** critical · **Team:** platform
|
||||
|
||||
## What it means
|
||||
|
||||
Prometheus scrapes all seven services through one address,
|
||||
`https://ledgerline.vione.cloud/metrics/<service>` (czy's Traefik, basic auth).
|
||||
"Down" means that scrape failed. It does **not** by itself prove the service is
|
||||
down: the service, the proxy, or the credentials can each cause it.
|
||||
|
||||
## First five minutes
|
||||
|
||||
1. **How many targets?** Open the alert in Alertmanager. One service alerting is
|
||||
likely that service. All seven at once points at the shared path (Traefik on
|
||||
czy, DNS, the scrape credentials), not at the services.
|
||||
2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and
|
||||
read its `Last error`.
|
||||
3. **Is the container running?** `ssh czy docker compose ps` for the service.
|
||||
4. **What did it last say?** `ssh czy docker logs ledgerline-<service>-1`, or in
|
||||
Loki: `{service_name="<service>"}`.
|
||||
5. **Is traffic still flowing?** Open the `RED: <service>` dashboard. Traces and
|
||||
logs can still be arriving while only the metrics scrape is broken.
|
||||
|
||||
## Escalate when
|
||||
|
||||
The shared path is down (all seven targets), or the service is down and not
|
||||
recovering after a restart.
|
||||
Reference in New Issue
Block a user