- ledgerline.transactions.total (counter: kind, status, currency) and
ledgerline.transaction.duration (histogram, seconds: kind) in gateway,
the only place that sees the final outcome, failures included. currency
is client input, so values outside the seeded set are recorded as
"other".
- ledgerline.fraud.score (histogram, buckets in 0.1 steps to 1.0) in fraud.
- ledgerline.stream.lag (observable gauge: consumer_group) in audit and
notifier, read from XINFO GROUPS through a separate sync Redis client,
because callbacks run on the scrape thread; the call is untraced so
scrapes don't create traces.
- ledgerline.db.pool.in_use (observable up-down counter: pool) in every
service with a DB engine. The SQLAlchemy instrumentation's own pool
metric names a pool by DSN, which merges the normal and starved engines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prose f-strings become past-tense events with consistent field names
across services (account_id, transaction_id, amount_minor, risk_score,
status_code, consumer_group, message_id, chaos_flag). The orchestrator
and both consumers move from logging.getLogger to structlog so they can
pass fields.
New events cover what was silent before: the gateway's UpstreamFailure
path (transaction_failed, with the underlying cause), fraud's score and
sanctions' screening outcomes, and each successfully processed stream
message. holder_name is no longer logged at all.
"lookup" and "healthz" still appear in exactly one event each, so the
pathology tests' log-line counts are unchanged until they move to LogQL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 6: downstream_client() no longer sends X-Request-Id, and request_id
is gone from the internal request bodies, models, function signatures
and log lines; correlation now travels as traceparent. The gateway's
public request_id stays (the contract tests require it) but is now the
trace ID, 32 hex digits and still a valid UUID, so the value the
frontend shows can be pasted into Tempo.
Step 7: delete tests/test_no_observability.py. It guarded the Phase 0
"no observability" state, which Phase 3 ends by design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where an error is caught and logged under a span that would otherwise
stay clean, also record_exception and set ERROR on that span: the
CONSUMER span when a stream message fails, fraud.score_transaction when
the sanctions call times out or fails, and the SERVER span of every
/readyz check (a 503 already made it ERROR, but without the exception).
Left alone where instrumentation already records it: ledger's app-wide
exception handler (FastAPI SERVER span), caught httpx errors (CLIENT
span), and the stream publish failure (PRODUCER span). The consumers'
failed reads run outside any span, so they stay log-only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual INTERNAL spans for the business steps the auto-instrumentation
can't name: fraud.score_transaction, sanctions.lookup_watchlist,
ledger.settle and insights.aggregate_120d, with ledgerline.* attributes
(account, amount, risk score, match outcome). holder_name stays off the
spans. Each span starts after its handler's chaos flag reads and
injection points, and no chaos-derived values become attributes.
The gateway's /api/health/all fan-out now runs under
suppress_instrumentation: with the route excluded, its five downstream
/healthz calls were each a parentless root trace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sanctions, notifier, ledger, insights and audit now call setup_telemetry
and instrument their engines with one SQLAlchemyInstrumentor call per
process (a second instrument() call is ignored, leaving that engine with
connect spans only). ledger and insights pass both their normal and
starved engines. The FastAPI services also get the server span, with the
same /healthz and receive/send exclusions as sanctions.
Compose: OTEL_SEMCONV_STABILITY_OPT_IN=http,database for the stable
attribute names, the OTLP endpoint moves to the shared env, and the four
newly instrumented services get the auth header from .env.observability.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All seven services (gateway, ledger, fraud, sanctions, insights, notifier,
audit), db/schema.sql plus optional_indexes.sql, scripts/seed.py and
smoke.py, the gateway-served frontend, and docker-compose.yml. Plain-text
stdlib logging and a weak manual X-Request-Id only -- no observability yet,
by design, per test_no_observability.py. Implements all 13 named chaos-flag
pathologies from docs/PATHOLOGIES.md behind Redis-backed runtime flags.
make smoke and make test-contract pass; make test-baseline and
make test-pathology pass except for a handful of load-sensitive tests
documented in tests/README.md rather than papered over.
Movses AslanyanandClaude Opus 5.5