Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
6.9 KiB
Pathology catalogue
Thirteen deliberate pathologies, toggled at runtime via the chaos:flags Redis
hash (never a restart). Each lives in a chaos.py module inside its owning
service. All flags default off.
Spoiler note (see root
CLAUDE.md): this file is a complete answer key for the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you, until that phase is done.
slow_sanctions (sanctions)
Mechanism: services/sanctions/chaos.py. On every /internal/lookup
call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of
calls instead sleep a flat 5s -- longer than the 3s downstream timeout every
caller uses (common/http.py), so these surface as timeouts one hop up, in
fraud.
Symptom: heavy-tailed latency on POST /api/transactions; most requests
look normal, a fat p95/p99 tail; occasional 504s from fraud/gateway with
no corresponding 5xx logged in sanctions itself (it never errors -- it's
just slow).
sanctions_errors (sanctions)
Mechanism: services/sanctions/chaos.py. 3% of /internal/lookup calls
return HTTP 503 outright, chosen independently per call via random.random().
Symptom: a small, steady rate of 502/504 responses from gateway, fully
uncorrelated with load or account. sanctions logs a "lookup" line for every
attempt regardless, so its request count won't match its success count.
n_plus_one (insights)
Mechanism: services/insights/analytics.py::_n_plus_one_summary. Instead
of one LEFT JOIN against merchant_categories, fetches up to 500 raw
transaction rows and issues one additional SELECT ... FROM merchant_categories WHERE id = :id per row.
Symptom: GET /api/accounts/{id}/insights for a high-volume (hot) account
issues 50-500+ SQL statements instead of 1-2, and its latency grows with the
account's transaction count rather than staying flat.
pool_starvation (ledger)
Mechanism: services/ledger/db.py. Two engines are built at startup: a
normal one (pool_size=20, max_overflow=10) and a starved one (pool_size=2, max_overflow=0, pool_timeout=10). Every ledger request picks which
already-running pool to borrow a connection from based on the current flag --
this is what makes the toggle live without a restart, since SQLAlchemy pools
can't be resized in place.
Symptom: throughput plateaus as concurrency increases past ~2 in-flight requests; latency and queueing rise sharply under concurrent load while a single request in isolation still looks fine.
row_lock_contention (ledger)
Mechanism: services/ledger/repository.py::settle_transaction. Only for
accounts with is_hot = true: takes SELECT balance_minor FROM accounts ... FOR UPDATE and holds the transaction open for an extra 150ms before
committing the balance update.
Symptom: concurrent transactions against a hot account show p95 latency several times higher than the same concurrency spread across many cold accounts; cold-account traffic is unaffected.
blocking_event_loop (fraud)
Mechanism: services/fraud/chaos.py::synchronous_cpu_work. Runs ~120ms of
synchronous SHA-256 hashing directly inside the async def handler for
/internal/score -- no run_in_executor/to_thread.
Symptom: concurrent requests to fraud (and anything waiting on it)
serialise: wall-clock time for N concurrent requests approaches N × single-request
time instead of staying close to single-request time, even though nothing is
CPU-bound at the infrastructure level.
retry_storm (gateway)
Mechanism: services/gateway/orchestrator.py::_score_with_retry. On any
non-2xx from fraud, retries immediately up to 3 more times -- no backoff,
no jitter, no circuit breaker.
Symptom: when sanctions is also failing or slow, the number of
sanctions "lookup" log lines per transaction multiplies (up to 4x), and
downstream load spikes exactly when the downstream is already unhealthy --
the opposite of what you want.
no_batching (gateway)
Mechanism: services/gateway/orchestrator.py::create_transaction. Splits
the transaction into 1-5 synthetic "line items" and calls fraud once per
item, sequentially, instead of once for the whole transaction.
Symptom: sanctions lookups per transaction multiply (roughly 2-5x) and
end-to-end latency for POST /api/transactions grows proportionally, even
though sanctions/fraud themselves are healthy.
unbounded_cache (insights)
Mechanism: services/insights/chaos.py / app.py. Instead of the normal
Redis cache (keyed by account + current chaos-flag fingerprint, TTL below),
memoizes summaries in a module-level dict keyed by (account_id, fingerprint, time.time_ns()) -- a key that is different on every single call. It never
hits, and it never evicts.
Symptom: insights' resident memory grows monotonically under repeated
identical-looking requests and never plateaus or drops; cache hit rate (if you
were watching Redis) would be zero -- the "cache" provides no speedup, only
a leak.
slow_consumer (notifier)
Mechanism: services/notifier/consumer.py. Sleeps 250ms per stream
message before acking, inside the notifier consumer-group loop.
Symptom: the notifier consumer-group lag on stream:transactions grows
under sustained load and drains slowly afterward, while the audit group
(same stream, independent group) keeps up fine -- isolating the problem to
one specific consumer, not the stream or producer.
cold_cache (insights)
Mechanism: services/insights/app.py. The Redis summary cache TTL drops
from 900s to 60s.
Symptom: repeated identical requests for the same account show periodic latency spikes roughly every 60s (a full recompute) instead of one initial spike followed by uniformly fast cache hits.
poison_data (ledger)
Mechanism: services/ledger/app.py / repository.py::find_null_currency_row.
Every POST /internal/transactions checks whether the account has any
existing transaction with a NULL currency (one of the seeded malformed row
varieties, §8). When the flag is on, that row's currency is accessed with no
null guard (.upper() on None), which raises and surfaces as an unhandled
HTTP 500. When off, the same check is guarded and does nothing.
Symptom: a specific, small set of accounts reliably 500 on every transaction attempt while the flag is on, and behave completely normally while it's off -- input data that was always there, suddenly fatal.
chatty_health (gateway)
Mechanism: services/gateway/orchestrator.py::_run_chatty_health_probe.
Before doing any real work, POST /api/transactions calls GET /healthz
sequentially on all five downstreams, tagged with an X-Ledgerline-Probe
header so the extra hits are distinguishable from routine health polling.
Symptom: every transaction request produces five extra, sequential health-check round trips before the real work even starts, inflating latency by a fixed per-request tax that has nothing to do with load or data size.