30 lines
1.3 KiB
Markdown
30 lines
1.3 KiB
Markdown
 
|
# LedgerlineTargetDown
|
||
|
|
|
||
|
|
**Fires when:** Prometheus has failed to scrape a Ledgerline service for 2 minutes.
|
||
|
|
**Severity:** critical · **Team:** platform
|
||
|
|
|
||
|
|
## What it means
|
||
|
|
|
||
|
|
Prometheus scrapes all seven services through one address,
|
||
|
|
`https://ledgerline.vione.cloud/metrics/<service>` (czy's Traefik, basic auth).
|
||
|
|
"Down" means that scrape failed. It does **not** by itself prove the service is
|
||
|
|
down: the service, the proxy, or the credentials can each cause it.
|
||
|
|
|
||
|
|
## First five minutes
|
||
|
|
|
||
|
|
1. **How many targets?** Open the alert in Alertmanager. One service alerting is
|
||
|
|
likely that service. All seven at once points at the shared path (Traefik on
|
||
|
|
czy, DNS, the scrape credentials), not at the services.
|
||
|
|
2. **Ask Prometheus why.** In Prometheus → Status → Targets, find the target and
|
||
|
|
read its `Last error`.
|
||
|
|
3. **Is the container running?** `ssh czy docker compose ps` for the service.
|
||
|
|
4. **What did it last say?** `ssh czy docker logs ledgerline-<service>-1`, or in
|
||
|
|
Loki: `{service_name="<service>"}`.
|
||
|
|
5. **Is traffic still flowing?** Open the `RED: <service>` dashboard. Traces and
|
||
|
|
logs can still be arriving while only the metrics scrape is broken.
|
||
|
|
|
||
|
|
## Escalate when
|
||
|
|
|
||
|
|
The shared path is down (all seven targets), or the service is down and not
|
||
|
|
recovering after a restart.
|