This reverts commit a1f1d80. Phase 5 step 4 is done: the burst minted
68 series across 60 account_id values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 5 step 4: a deliberate cardinality explosion. Every account that
transacts mints its own series. To be reverted after the burst.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ledgerline.transactions.total (counter: kind, status, currency) and
ledgerline.transaction.duration (histogram, seconds: kind) in gateway,
the only place that sees the final outcome, failures included. currency
is client input, so values outside the seeded set are recorded as
"other".
- ledgerline.fraud.score (histogram, buckets in 0.1 steps to 1.0) in fraud.
- ledgerline.stream.lag (observable gauge: consumer_group) in audit and
notifier, read from XINFO GROUPS through a separate sync Redis client,
because callbacks run on the scrape thread; the call is untraced so
scrapes don't create traces.
- ledgerline.db.pool.in_use (observable up-down counter: pool) in every
service with a DB engine. The SQLAlchemy instrumentation's own pool
metric names a pool by DSN, which merges the normal and starved engines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prose f-strings become past-tense events with consistent field names
across services (account_id, transaction_id, amount_minor, risk_score,
status_code, consumer_group, message_id, chaos_flag). The orchestrator
and both consumers move from logging.getLogger to structlog so they can
pass fields.
New events cover what was silent before: the gateway's UpstreamFailure
path (transaction_failed, with the underlying cause), fraud's score and
sanctions' screening outcomes, and each successfully processed stream
message. holder_name is no longer logged at all.
"lookup" and "healthz" still appear in exactly one event each, so the
pathology tests' log-line counts are unchanged until they move to LogQL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 6: downstream_client() no longer sends X-Request-Id, and request_id
is gone from the internal request bodies, models, function signatures
and log lines; correlation now travels as traceparent. The gateway's
public request_id stays (the contract tests require it) but is now the
trace ID, 32 hex digits and still a valid UUID, so the value the
frontend shows can be pasted into Tempo.
Step 7: delete tests/test_no_observability.py. It guarded the Phase 0
"no observability" state, which Phase 3 ends by design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual INTERNAL spans for the business steps the auto-instrumentation
can't name: fraud.score_transaction, sanctions.lookup_watchlist,
ledger.settle and insights.aggregate_120d, with ledgerline.* attributes
(account, amount, risk score, match outcome). holder_name stays off the
spans. Each span starts after its handler's chaos flag reads and
injection points, and no chaos-derived values become attributes.
The gateway's /api/health/all fan-out now runs under
suppress_instrumentation: with the route excluded, its five downstream
/healthz calls were each a parentless root trace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redis Streams have no headers, so the gateway injects traceparent into
the message fields inside a PRODUCER span. The notifier and audit
consumers extract it and open one CONSUMER span per message, so their
DB and XACK spans join the transaction's trace. The idle XREADGROUP poll
runs under suppress_instrumentation (it was a ~2s root trace every loop),
and audit strips the trace fields so its stored payload is unchanged.
The gateway also stops tracing its frontend's /api/health/ polling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setup_telemetry now instruments httpx and redis process-wide. Both patch
the client classes, so clients created at import time (gateway's
_proxy_client, common.redis_client) are covered too. httpx CLIENT spans
inject traceparent, so the downstream FastAPI SERVER spans join the
caller's trace. gateway and fraud get setup_telemetry, the FastAPI span,
and the OTLP auth header. All seven services now export traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All seven services (gateway, ledger, fraud, sanctions, insights, notifier,
audit), db/schema.sql plus optional_indexes.sql, scripts/seed.py and
smoke.py, the gateway-served frontend, and docker-compose.yml. Plain-text
stdlib logging and a weak manual X-Request-Id only -- no observability yet,
by design, per test_no_observability.py. Implements all 13 named chaos-flag
pathologies from docs/PATHOLOGIES.md behind Redis-backed runtime flags.
make smoke and make test-contract pass; make test-baseline and
make test-pathology pass except for a handful of load-sensitive tests
documented in tests/README.md rather than papered over.
Movses AslanyanandClaude Opus 5.5