Record swallowed exceptions on their spans

Where an error is caught and logged under a span that would otherwise
stay clean, also record_exception and set ERROR on that span: the
CONSUMER span when a stream message fails, fraud.score_transaction when
the sanctions call times out or fails, and the SERVER span of every
/readyz check (a 503 already made it ERROR, but without the exception).

Left alone where instrumentation already records it: ledger's app-wide
exception handler (FastAPI SERVER span), caught httpx errors (CLIENT
span), and the stream publish failure (PRODUCER span). The consumers'
failed reads run outside any span, so they stay log-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-13 19:48:27 +03:30
co-authored by Claude Opus 5.5
parent d52a19965a
commit 1ab425dc44
7 changed files with 37 additions and 7 deletions
+5 -3
View File
@@ -6,7 +6,7 @@ import uuid
import sqlalchemy as sa
from opentelemetry import propagate, trace
from opentelemetry.instrumentation.utils import suppress_instrumentation
from opentelemetry.trace import SpanKind
from opentelemetry.trace import SpanKind, StatusCode
from common.db import Session
from common.redis_client import redis_client
@@ -89,9 +89,11 @@ async def run():
"messaging.consumer.group.name": GROUP,
"messaging.message.id": msg_id,
},
):
) as span:
try:
await _handle(fields)
except Exception:
except Exception as exc:
span.record_exception(exc)
span.set_status(StatusCode.ERROR, str(exc))
logger.exception(f"notifier failed to process message {msg_id}")
await redis_client.xack(settings.stream_name, GROUP, msg_id)