Movses AslanyanandClaude Sonnet 5.5
fef735c922
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1.3 KiB
1.3 KiB
LedgerlineTargetDown
Fires when: Prometheus has failed to scrape a Ledgerline service for 2 minutes. Severity: critical · Team: platform
What it means
Prometheus scrapes all seven services through one address,
https://ledgerline.vione.cloud/metrics/<service> (czy's Traefik, basic auth).
"Down" means that scrape failed. It does not by itself prove the service is
down: the service, the proxy, or the credentials can each cause it.
First five minutes
- How many targets? Open the alert in Alertmanager. One service alerting is likely that service. All seven at once points at the shared path (Traefik on czy, DNS, the scrape credentials), not at the services.
- Ask Prometheus why. In Prometheus → Status → Targets, find the target and
read its
Last error. - Is the container running?
ssh czy docker compose psfor the service. - What did it last say?
ssh czy docker logs ledgerline-<service>-1, or in Loki:{service_name="<service>"}. - Is traffic still flowing? Open the
RED: <service>dashboard. Traces and logs can still be arriving while only the metrics scrape is broken.
Escalate when
The shared path is down (all seven targets), or the service is down and not recovering after a restart.