28 lines
1.2 KiB
Markdown
28 lines
1.2 KiB
Markdown
 
|
# LedgerlineHighErrorRatio
|
||
|
|
|
||
|
|
**Fires when:** a service's 5xx ratio is above 5% for 5 minutes, with more than
|
||
|
|
0.05 requests/s (so it is not one failed request on an idle service).
|
||
|
|
**Severity:** warning · **Team:** payments
|
||
|
|
|
||
|
|
## What it means
|
||
|
|
|
||
|
|
A meaningful share of the requests this service serves are failing with a server
|
||
|
|
error. 4xx responses (for example a 404) are deliberately not counted.
|
||
|
|
|
||
|
|
## First five minutes
|
||
|
|
|
||
|
|
1. **Look at the `RED: <service>` dashboard** (linked in the alert). Is the error
|
||
|
|
ratio still elevated, or already back to normal? Did the request rate or the
|
||
|
|
latency change at the same time?
|
||
|
|
2. **Find a failing request.** In Tempo, search for the service with an error
|
||
|
|
status, or in Loki: `{service_name="<service>"} | json | level="ERROR"`.
|
||
|
|
3. **Follow one trace.** The `trace_id` in the log line opens the whole request
|
||
|
|
across services. Look for the first span in the trace with an error status.
|
||
|
|
4. **Is it only this service?** The System overview's service graph shows whether
|
||
|
|
its callers or callees are failing as well.
|
||
|
|
|
||
|
|
## Escalate when
|
||
|
|
|
||
|
|
The ratio is still rising after ten minutes, or the failing requests are on the
|
||
|
|
critical path (transactions), not a side route.
|