Add dashboards and alerts as code, mark Phase 6 done
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5.5
parent
b50f8c3b24
commit
fef735c922
@@ -0,0 +1,27 @@
|
||||
# LedgerlineHighErrorRatio
|
||||
|
||||
**Fires when:** a service's 5xx ratio is above 5% for 5 minutes, with more than
|
||||
0.05 requests/s (so it is not one failed request on an idle service).
|
||||
**Severity:** warning · **Team:** payments
|
||||
|
||||
## What it means
|
||||
|
||||
A meaningful share of the requests this service serves are failing with a server
|
||||
error. 4xx responses (for example a 404) are deliberately not counted.
|
||||
|
||||
## First five minutes
|
||||
|
||||
1. **Look at the `RED: <service>` dashboard** (linked in the alert). Is the error
|
||||
ratio still elevated, or already back to normal? Did the request rate or the
|
||||
latency change at the same time?
|
||||
2. **Find a failing request.** In Tempo, search for the service with an error
|
||||
status, or in Loki: `{service_name="<service>"} | json | level="ERROR"`.
|
||||
3. **Follow one trace.** The `trace_id` in the log line opens the whole request
|
||||
across services. Look for the first span in the trace with an error status.
|
||||
4. **Is it only this service?** The System overview's service graph shows whether
|
||||
its callers or callees are failing as well.
|
||||
|
||||
## Escalate when
|
||||
|
||||
The ratio is still rising after ten minutes, or the failing requests are on the
|
||||
critical path (transactions), not a side route.
|
||||
Reference in New Issue
Block a user