Movses AslanyanandClaude Sonnet 5.5
fef735c922
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
28 lines
1.2 KiB
Markdown
28 lines
1.2 KiB
Markdown
# LedgerlineHighErrorRatio
|
|
|
|
**Fires when:** a service's 5xx ratio is above 5% for 5 minutes, with more than
|
|
0.05 requests/s (so it is not one failed request on an idle service).
|
|
**Severity:** warning · **Team:** payments
|
|
|
|
## What it means
|
|
|
|
A meaningful share of the requests this service serves are failing with a server
|
|
error. 4xx responses (for example a 404) are deliberately not counted.
|
|
|
|
## First five minutes
|
|
|
|
1. **Look at the `RED: <service>` dashboard** (linked in the alert). Is the error
|
|
ratio still elevated, or already back to normal? Did the request rate or the
|
|
latency change at the same time?
|
|
2. **Find a failing request.** In Tempo, search for the service with an error
|
|
status, or in Loki: `{service_name="<service>"} | json | level="ERROR"`.
|
|
3. **Follow one trace.** The `trace_id` in the log line opens the whole request
|
|
across services. Look for the first span in the trace with an error status.
|
|
4. **Is it only this service?** The System overview's service graph shows whether
|
|
its callers or callees are failing as well.
|
|
|
|
## Escalate when
|
|
|
|
The ratio is still rising after ten minutes, or the failing requests are on the
|
|
critical path (transactions), not a side route.
|