- ledgerline.transactions.total (counter: kind, status, currency) and
ledgerline.transaction.duration (histogram, seconds: kind) in gateway,
the only place that sees the final outcome, failures included. currency
is client input, so values outside the seeded set are recorded as
"other".
- ledgerline.fraud.score (histogram, buckets in 0.1 steps to 1.0) in fraud.
- ledgerline.stream.lag (observable gauge: consumer_group) in audit and
notifier, read from XINFO GROUPS through a separate sync Redis client,
because callbacks run on the scrape thread; the call is untraced so
scrapes don't create traces.
- ledgerline.db.pool.in_use (observable up-down counter: pool) in every
service with a DB engine. The SQLAlchemy instrumentation's own pool
metric names a pool by DSN, which merges the normal and starved engines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alloy 404s OTLP metrics, so each service gets a MeterProvider with a
PrometheusMetricReader (same Resource as traces) served by
prometheus_client on its own port instead of a FastAPI route: gateway's
port 8000 is publicly routed by Traefik, and the audit worker has no HTTP
app. Host/port come from OTEL_EXPORTER_PROMETHEUS_HOST/PORT. Setting the
provider before the instrumentors also turns on their built-in HTTP and
DB pool metrics.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A processor in the shared chain masks sensitive field names (credential
fragments like password/token/authorization, plus exact names such as
card_number and holder_name) at any depth, for structlog and stdlib
records alike. It sees field names only: values interpolated into the
event string still leak until call sites pass them as fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A processor in the shared chain reads the active span context and adds
its IDs as lowercase hex, so both structlog and stdlib records carry
them. The lowercase `trace_id` name is what the provisioned Loki derived
field's case-sensitive regex matches; outside a span nothing is added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
configure_logging now builds one processor chain shared by structlog
calls and foreign stdlib records (uvicorn, the consumers' getLogger),
rendered as one JSON object per line on stdout. Tracebacks stay inside
their line instead of splitting into one Docker log entry per frame.
uvicorn.access is deliberately left untouched: --no-access-log works by
disabling its propagation, and re-enabling it would bring access logs
back (and inflate the healthz/lookup line counts the tests rely on).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 6: downstream_client() no longer sends X-Request-Id, and request_id
is gone from the internal request bodies, models, function signatures
and log lines; correlation now travels as traceparent. The gateway's
public request_id stays (the contract tests require it) but is now the
trace ID, 32 hex digits and still a valid UUID, so the value the
frontend shows can be pasted into Tempo.
Step 7: delete tests/test_no_observability.py. It guarded the Phase 0
"no observability" state, which Phase 3 ends by design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setup_telemetry now instruments httpx and redis process-wide. Both patch
the client classes, so clients created at import time (gateway's
_proxy_client, common.redis_client) are covered too. httpx CLIENT spans
inject traceparent, so the downstream FastAPI SERVER spans join the
caller's trace. gateway and fraud get setup_telemetry, the FastAPI span,
and the OTLP auth header. All seven services now export traces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
common/telemetry.py builds the Resource, TracerProvider, BatchSpanProcessor
and OTLP/HTTP exporter by hand; sanctions calls setup_telemetry() at
import instead of running under opentelemetry-instrument. No instrumentor
is enabled yet, so sanctions emits no spans for now.
service.version comes from OTEL_RESOURCE_ATTRIBUTES: deploy.sh exports
DEPLOY_HASH (the server's HEAD after the pull) and compose falls back to
"dev". deployment.environment.name=prod is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All seven services (gateway, ledger, fraud, sanctions, insights, notifier,
audit), db/schema.sql plus optional_indexes.sql, scripts/seed.py and
smoke.py, the gateway-served frontend, and docker-compose.yml. Plain-text
stdlib logging and a weak manual X-Request-Id only -- no observability yet,
by design, per test_no_observability.py. Implements all 13 named chaos-flag
pathologies from docs/PATHOLOGIES.md behind Redis-backed runtime flags.
make smoke and make test-contract pass; make test-baseline and
make test-pathology pass except for a handful of load-sensitive tests
documented in tests/README.md rather than papered over.
Movses AslanyanandClaude Opus 5.5