# Pathology catalogue

Thirteen deliberate pathologies, toggled at runtime via the `chaos:flags` Redis
hash (never a restart). Each lives in a `chaos.py` module inside its owning
service. All flags default off.

> Spoiler note (see root `CLAUDE.md`): this file is a complete answer key for
> the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you,
> until that phase is done.

---

## slow_sanctions (`sanctions`)

**Mechanism:** `services/sanctions/chaos.py`. On every `/internal/lookup`
call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of
calls instead sleep a flat 5s -- longer than the 3s downstream timeout every
caller uses (`common/http.py`), so these surface as timeouts one hop up, in
`fraud`.

**Symptom:** heavy-tailed latency on `POST /api/transactions`; most requests
look normal, a fat p95/p99 tail; occasional 504s from `fraud`/`gateway` with
no corresponding 5xx logged in `sanctions` itself (it never errors -- it's
just slow).

## sanctions_errors (`sanctions`)

**Mechanism:** `services/sanctions/chaos.py`. 3% of `/internal/lookup` calls
return HTTP 503 outright, chosen independently per call via `random.random()`.

**Symptom:** a small, steady rate of 502/504 responses from `gateway`, fully
uncorrelated with load or account. `sanctions` logs a "lookup" line for every
attempt regardless, so its request count won't match its success count.

## n_plus_one (`insights`)

**Mechanism:** `services/insights/analytics.py::_n_plus_one_summary`. Instead
of one `LEFT JOIN` against `merchant_categories`, fetches up to 500 raw
transaction rows and issues one additional `SELECT ... FROM merchant_categories
WHERE id = :id` per row.

**Symptom:** `GET /api/accounts/{id}/insights` for a high-volume (hot) account
issues 50-500+ SQL statements instead of 1-2, and its latency grows with the
account's transaction count rather than staying flat.

## pool_starvation (`ledger`)

**Mechanism:** `services/ledger/db.py`. Two engines are built at startup: a
normal one (`pool_size=20, max_overflow=10`) and a starved one (`pool_size=2,
max_overflow=0, pool_timeout=10`). Every `ledger` request picks which
already-running pool to borrow a connection from based on the current flag --
this is what makes the toggle live without a restart, since SQLAlchemy pools
can't be resized in place.

**Symptom:** throughput plateaus as concurrency increases past ~2 in-flight
requests; latency and queueing rise sharply under concurrent load while a
single request in isolation still looks fine.

## row_lock_contention (`ledger`)

**Mechanism:** `services/ledger/repository.py::settle_transaction`. Only for
accounts with `is_hot = true`: takes `SELECT balance_minor FROM accounts ...
FOR UPDATE` and holds the transaction open for an extra 150ms before
committing the balance update.

**Symptom:** concurrent transactions against a hot account show p95 latency
several times higher than the same concurrency spread across many cold
accounts; cold-account traffic is unaffected.

## blocking_event_loop (`fraud`)

**Mechanism:** `services/fraud/chaos.py::synchronous_cpu_work`. Runs ~120ms of
synchronous SHA-256 hashing directly inside the `async def` handler for
`/internal/score` -- no `run_in_executor`/`to_thread`.

**Symptom:** concurrent requests to `fraud` (and anything waiting on it)
serialise: wall-clock time for N concurrent requests approaches N × single-request
time instead of staying close to single-request time, even though nothing is
CPU-bound at the infrastructure level.

## retry_storm (`gateway`)

**Mechanism:** `services/gateway/orchestrator.py::_score_with_retry`. On any
non-2xx from `fraud`, retries immediately up to 3 more times -- no backoff,
no jitter, no circuit breaker.

**Symptom:** when `sanctions` is also failing or slow, the number of
`sanctions` "lookup" log lines per transaction multiplies (up to 4x), and
downstream load spikes exactly when the downstream is already unhealthy --
the opposite of what you want.

## no_batching (`gateway`)

**Mechanism:** `services/gateway/orchestrator.py::create_transaction`. Splits
the transaction into 1-5 synthetic "line items" and calls `fraud` once per
item, sequentially, instead of once for the whole transaction.

**Symptom:** `sanctions` lookups per transaction multiply (roughly 2-5x) and
end-to-end latency for `POST /api/transactions` grows proportionally, even
though sanctions/fraud themselves are healthy.

## unbounded_cache (`insights`)

**Mechanism:** `services/insights/chaos.py` / `app.py`. Instead of the normal
Redis cache (keyed by account + current chaos-flag fingerprint, TTL below),
memoizes summaries in a module-level dict keyed by `(account_id, fingerprint,
time.time_ns())` -- a key that is different on every single call. It never
hits, and it never evicts.

**Symptom:** `insights`' resident memory grows monotonically under repeated
identical-looking requests and never plateaus or drops; cache hit rate (if you
were watching Redis) would be zero -- the "cache" provides no speedup, only
a leak.

## slow_consumer (`notifier`)

**Mechanism:** `services/notifier/consumer.py`. Sleeps 250ms per stream
message before acking, inside the `notifier` consumer-group loop.

**Symptom:** the `notifier` consumer-group lag on `stream:transactions` grows
under sustained load and drains slowly afterward, while the `audit` group
(same stream, independent group) keeps up fine -- isolating the problem to
one specific consumer, not the stream or producer.

## cold_cache (`insights`)

**Mechanism:** `services/insights/app.py`. The Redis summary cache TTL drops
from 900s to 60s.

**Symptom:** repeated identical requests for the same account show periodic
latency spikes roughly every 60s (a full recompute) instead of one initial
spike followed by uniformly fast cache hits.

## poison_data (`ledger`)

**Mechanism:** `services/ledger/app.py` / `repository.py::find_null_currency_row`.
Every `POST /internal/transactions` checks whether the account has any
existing transaction with a `NULL` currency (one of the seeded malformed row
varieties, §8). When the flag is on, that row's `currency` is accessed with no
null guard (`.upper()` on `None`), which raises and surfaces as an unhandled
HTTP 500. When off, the same check is guarded and does nothing.

**Symptom:** a specific, small set of accounts reliably 500 on every
transaction attempt while the flag is on, and behave completely normally
while it's off -- input data that was always there, suddenly fatal.

## chatty_health (`gateway`)

**Mechanism:** `services/gateway/orchestrator.py::_run_chatty_health_probe`.
Before doing any real work, `POST /api/transactions` calls `GET /healthz`
sequentially on all five downstreams, tagged with an `X-Ledgerline-Probe`
header so the extra hits are distinguishable from routine health polling.

**Symptom:** every transaction request produces five extra, sequential
health-check round trips before the real work even starts, inflating latency
by a fixed per-request tax that has nothing to do with load or data size.