Movses AslanyanandClaude Sonnet 5.5
fef735c922
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1.2 KiB
1.2 KiB
LedgerlineHighErrorRatio
Fires when: a service's 5xx ratio is above 5% for 5 minutes, with more than 0.05 requests/s (so it is not one failed request on an idle service). Severity: warning · Team: payments
What it means
A meaningful share of the requests this service serves are failing with a server error. 4xx responses (for example a 404) are deliberately not counted.
First five minutes
- Look at the
RED: <service>dashboard (linked in the alert). Is the error ratio still elevated, or already back to normal? Did the request rate or the latency change at the same time? - Find a failing request. In Tempo, search for the service with an error
status, or in Loki:
{service_name="<service>"} | json | level="ERROR". - Follow one trace. The
trace_idin the log line opens the whole request across services. Look for the first span in the trace with an error status. - Is it only this service? The System overview's service graph shows whether its callers or callees are failing as well.
Escalate when
The ratio is still rising after ten minutes, or the failing requests are on the critical path (transactions), not a side route.