Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
157 lines
6.9 KiB
Markdown
157 lines
6.9 KiB
Markdown
# Pathology catalogue
|
||
|
||
Thirteen deliberate pathologies, toggled at runtime via the `chaos:flags` Redis
|
||
hash (never a restart). Each lives in a `chaos.py` module inside its owning
|
||
service. All flags default off.
|
||
|
||
> Spoiler note (see root `CLAUDE.md`): this file is a complete answer key for
|
||
> the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you,
|
||
> until that phase is done.
|
||
|
||
---
|
||
|
||
## slow_sanctions (`sanctions`)
|
||
|
||
**Mechanism:** `services/sanctions/chaos.py`. On every `/internal/lookup`
|
||
call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of
|
||
calls instead sleep a flat 5s -- longer than the 3s downstream timeout every
|
||
caller uses (`common/http.py`), so these surface as timeouts one hop up, in
|
||
`fraud`.
|
||
|
||
**Symptom:** heavy-tailed latency on `POST /api/transactions`; most requests
|
||
look normal, a fat p95/p99 tail; occasional 504s from `fraud`/`gateway` with
|
||
no corresponding 5xx logged in `sanctions` itself (it never errors -- it's
|
||
just slow).
|
||
|
||
## sanctions_errors (`sanctions`)
|
||
|
||
**Mechanism:** `services/sanctions/chaos.py`. 3% of `/internal/lookup` calls
|
||
return HTTP 503 outright, chosen independently per call via `random.random()`.
|
||
|
||
**Symptom:** a small, steady rate of 502/504 responses from `gateway`, fully
|
||
uncorrelated with load or account. `sanctions` logs a "lookup" line for every
|
||
attempt regardless, so its request count won't match its success count.
|
||
|
||
## n_plus_one (`insights`)
|
||
|
||
**Mechanism:** `services/insights/analytics.py::_n_plus_one_summary`. Instead
|
||
of one `LEFT JOIN` against `merchant_categories`, fetches up to 500 raw
|
||
transaction rows and issues one additional `SELECT ... FROM merchant_categories
|
||
WHERE id = :id` per row.
|
||
|
||
**Symptom:** `GET /api/accounts/{id}/insights` for a high-volume (hot) account
|
||
issues 50-500+ SQL statements instead of 1-2, and its latency grows with the
|
||
account's transaction count rather than staying flat.
|
||
|
||
## pool_starvation (`ledger`)
|
||
|
||
**Mechanism:** `services/ledger/db.py`. Two engines are built at startup: a
|
||
normal one (`pool_size=20, max_overflow=10`) and a starved one (`pool_size=2,
|
||
max_overflow=0, pool_timeout=10`). Every `ledger` request picks which
|
||
already-running pool to borrow a connection from based on the current flag --
|
||
this is what makes the toggle live without a restart, since SQLAlchemy pools
|
||
can't be resized in place.
|
||
|
||
**Symptom:** throughput plateaus as concurrency increases past ~2 in-flight
|
||
requests; latency and queueing rise sharply under concurrent load while a
|
||
single request in isolation still looks fine.
|
||
|
||
## row_lock_contention (`ledger`)
|
||
|
||
**Mechanism:** `services/ledger/repository.py::settle_transaction`. Only for
|
||
accounts with `is_hot = true`: takes `SELECT balance_minor FROM accounts ...
|
||
FOR UPDATE` and holds the transaction open for an extra 150ms before
|
||
committing the balance update.
|
||
|
||
**Symptom:** concurrent transactions against a hot account show p95 latency
|
||
several times higher than the same concurrency spread across many cold
|
||
accounts; cold-account traffic is unaffected.
|
||
|
||
## blocking_event_loop (`fraud`)
|
||
|
||
**Mechanism:** `services/fraud/chaos.py::synchronous_cpu_work`. Runs ~120ms of
|
||
synchronous SHA-256 hashing directly inside the `async def` handler for
|
||
`/internal/score` -- no `run_in_executor`/`to_thread`.
|
||
|
||
**Symptom:** concurrent requests to `fraud` (and anything waiting on it)
|
||
serialise: wall-clock time for N concurrent requests approaches N × single-request
|
||
time instead of staying close to single-request time, even though nothing is
|
||
CPU-bound at the infrastructure level.
|
||
|
||
## retry_storm (`gateway`)
|
||
|
||
**Mechanism:** `services/gateway/orchestrator.py::_score_with_retry`. On any
|
||
non-2xx from `fraud`, retries immediately up to 3 more times -- no backoff,
|
||
no jitter, no circuit breaker.
|
||
|
||
**Symptom:** when `sanctions` is also failing or slow, the number of
|
||
`sanctions` "lookup" log lines per transaction multiplies (up to 4x), and
|
||
downstream load spikes exactly when the downstream is already unhealthy --
|
||
the opposite of what you want.
|
||
|
||
## no_batching (`gateway`)
|
||
|
||
**Mechanism:** `services/gateway/orchestrator.py::create_transaction`. Splits
|
||
the transaction into 1-5 synthetic "line items" and calls `fraud` once per
|
||
item, sequentially, instead of once for the whole transaction.
|
||
|
||
**Symptom:** `sanctions` lookups per transaction multiply (roughly 2-5x) and
|
||
end-to-end latency for `POST /api/transactions` grows proportionally, even
|
||
though sanctions/fraud themselves are healthy.
|
||
|
||
## unbounded_cache (`insights`)
|
||
|
||
**Mechanism:** `services/insights/chaos.py` / `app.py`. Instead of the normal
|
||
Redis cache (keyed by account + current chaos-flag fingerprint, TTL below),
|
||
memoizes summaries in a module-level dict keyed by `(account_id, fingerprint,
|
||
time.time_ns())` -- a key that is different on every single call. It never
|
||
hits, and it never evicts.
|
||
|
||
**Symptom:** `insights`' resident memory grows monotonically under repeated
|
||
identical-looking requests and never plateaus or drops; cache hit rate (if you
|
||
were watching Redis) would be zero -- the "cache" provides no speedup, only
|
||
a leak.
|
||
|
||
## slow_consumer (`notifier`)
|
||
|
||
**Mechanism:** `services/notifier/consumer.py`. Sleeps 250ms per stream
|
||
message before acking, inside the `notifier` consumer-group loop.
|
||
|
||
**Symptom:** the `notifier` consumer-group lag on `stream:transactions` grows
|
||
under sustained load and drains slowly afterward, while the `audit` group
|
||
(same stream, independent group) keeps up fine -- isolating the problem to
|
||
one specific consumer, not the stream or producer.
|
||
|
||
## cold_cache (`insights`)
|
||
|
||
**Mechanism:** `services/insights/app.py`. The Redis summary cache TTL drops
|
||
from 900s to 60s.
|
||
|
||
**Symptom:** repeated identical requests for the same account show periodic
|
||
latency spikes roughly every 60s (a full recompute) instead of one initial
|
||
spike followed by uniformly fast cache hits.
|
||
|
||
## poison_data (`ledger`)
|
||
|
||
**Mechanism:** `services/ledger/app.py` / `repository.py::find_null_currency_row`.
|
||
Every `POST /internal/transactions` checks whether the account has any
|
||
existing transaction with a `NULL` currency (one of the seeded malformed row
|
||
varieties, §8). When the flag is on, that row's `currency` is accessed with no
|
||
null guard (`.upper()` on `None`), which raises and surfaces as an unhandled
|
||
HTTP 500. When off, the same check is guarded and does nothing.
|
||
|
||
**Symptom:** a specific, small set of accounts reliably 500 on every
|
||
transaction attempt while the flag is on, and behave completely normally
|
||
while it's off -- input data that was always there, suddenly fatal.
|
||
|
||
## chatty_health (`gateway`)
|
||
|
||
**Mechanism:** `services/gateway/orchestrator.py::_run_chatty_health_probe`.
|
||
Before doing any real work, `POST /api/transactions` calls `GET /healthz`
|
||
sequentially on all five downstreams, tagged with an `X-Ledgerline-Probe`
|
||
header so the extra hits are distinguishable from routine health polling.
|
||
|
||
**Symptom:** every transaction request produces five extra, sequential
|
||
health-check round trips before the real work even starts, inflating latency
|
||
by a fixed per-request tax that has nothing to do with load or data size.
|