Movses AslanyanandClaude Sonnet 5.5
fef735c922
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
30 lines
1.3 KiB
Markdown
30 lines
1.3 KiB
Markdown
# LedgerlineTargetDown
|
|
|
|
**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes.
|
|
**Severity:** critical · **Team:** platform
|
|
|
|
## What it means
|
|
|
|
Prometheus scrapes all seven services through one address,
|
|
`https://ledgerline.vione.cloud/metrics/<service>` (czy's Traefik, basic auth).
|
|
"Down" means that scrape failed. It does **not** by itself prove the service is
|
|
down: the service, the proxy, or the credentials can each cause it.
|
|
|
|
## First five minutes
|
|
|
|
1. **How many targets?** Open the alert in Alertmanager. One service alerting is
|
|
likely that service. All seven at once points at the shared path (Traefik on
|
|
czy, DNS, the scrape credentials), not at the services.
|
|
2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and
|
|
read its `Last error`.
|
|
3. **Is the container running?** `ssh czy docker compose ps` for the service.
|
|
4. **What did it last say?** `ssh czy docker logs ledgerline-<service>-1`, or in
|
|
Loki: `{service_name="<service>"}`.
|
|
5. **Is traffic still flowing?** Open the `RED: <service>` dashboard. Traces and
|
|
logs can still be arriving while only the metrics scrape is broken.
|
|
|
|
## Escalate when
|
|
|
|
The shared path is down (all seven targets), or the service is down and not
|
|
recovering after a restart.
|