LedgerLine/tests/README.md

# Ledgerline integration suite

Black-box, per `docs/phase-0a-test-suite-spec.md`. Talks to the system only
over HTTP, Postgres, Redis, and the Docker CLI -- no imports from `services/`
or `common/`. Once Phase 0b is green, treat this directory as a **read-only
contract**: no test, threshold, marker, or fixture gets edited to make code
pass. If a test looks wrong or impossible, stop and say so.

## Running it

Prerequisite for every marker: `make up && make seed`.

```bash
make test              # everything except slow (the default CI-shaped run)
make test-contract     # contract + seed + guard, < 60s
make test-baseline     # baseline, < 90s
make test-pathology    # pathology and not slow, 5-12 min
make test-slow         # slow (unbounded_cache, cold_cache), several minutes
make test-all          # no marker filter
```

> **Phase 8 spoiler warning** (see the root `CLAUDE.md`): `test-pathology` and
> `test-slow` are a complete answer key to which chaos flags do what. Don't
> run them during the Phase 8 bottleneck hunt.

Config is env-driven with Compose defaults as fallback -- see
`tests/harness/config.py` for the full list (`GATEWAY_URL`, `LEDGER_URL`, ...,
`POSTGRES_DSN`, `REDIS_URL`, `SEED`). Override any of them to point the suite
at a non-default stack.

## Request contract assumptions

Neither spec pins down the JSON body for `POST /api/transactions` -- only the
response envelope. Since this suite *is* the contract until Phase 0b exists,
`tests/harness/payloads.py::tx_payload()` fixes it:

```json
{
  "account_id": "<uuid>",
  "kind": "deposit" | "withdrawal" | "purchase",
  "amount_minor": <int>,
  "currency": "EUR" | "USD" | "GBP",
  "merchant_name": "<string>",
  "category_id": <smallint, optional>
}
```

Agent B's implementation must accept exactly this shape. If it turns out to
be wrong or incomplete once real requirements surface, that's a suite change
to negotiate explicitly -- not something to route around silently.

## Log-line assumptions

Three pathology tests (6.8, 6.9, 6.14) count container log lines, per
`phase-0a-test-suite-spec.md`'s note that this is the only cross-service call
counter available at Phase 0. They assume, and Agent B's plain-text logging
must satisfy:

- `sanctions` logs a line containing the word "lookup" (case-insensitive) for
  every watchlist check it performs.
- Every downstream service logs a line containing "healthz" (case-insensitive)
  when its health endpoint is hit.

See `tests/harness/config.py` for the exact patterns
(`SANCTIONS_LOOKUP_LOG_PATTERN`, `HEALTHZ_LOG_PATTERN`).

## Measurement methodology

- **Ratios, not absolutes.** With one deliberate exception (G7's p95 < 400ms,
  which the spec itself calls out as "the one absolute threshold in the
  functional set"), every assertion compares flag-on against flag-off in the
  same run, on the same machine. Absolute millisecond thresholds don't survive
  a different laptop; ratios mostly do.
- **Percentiles**: nearest-rank on the raw sorted sample, no interpolation.
  Never average percentiles across runs -- `harness.load.measure_median`
  picks the whole run whose key metric (e.g. p95) is the median, so you get a
  consistent report, not a Frankenstein average.
- **Prefer signals that are hard to fake.** `pg_stat_statements` call counts
  (6.3, and the live-reconfig check in `test_contract_chaos.py`), `EXPLAIN`
  plan node types (6.4), Redis consumer-group lag (6.11, and G7/G3),
  container RSS (6.10), and a genuine unhandled-exception path (6.13) can't be
  satisfied by a `sleep()`. Where both a hard signal and a latency ratio are
  available, both are asserted.
- **Warm-up requests are discarded** before timing starts, per `harness.load.drive`.

## Coordinated omission

`harness/load.py::drive` is a closed-loop driver: fixed concurrency, next
request issued only after a slot frees up. This is simple but
**under-measures latency under saturation** -- a slow response delays the
next request's start rather than being counted as more queueing time. Phase 6
replaces this with k6's arrival-rate (open-model) executors. This suite
doesn't try to fix it; the ratios it asserts are generous enough to tolerate
the distortion, but don't mistake the numbers here for a real load-testing
tool's output.

## Known-flaky tests

**6.10 (`unbounded_cache`) and 6.12 (`cold_cache`)** are the least stable in
the suite -- both marked `slow` and excluded from `make test`/`test-pathology`:

- 6.10 depends on `docker stats` RSS sampling being reasonably quiet on the
  host (no other memory-hungry process competing) across a 30s window twice.
  A genuine failure looks like *no* growth difference between baseline and
  degraded (the leak isn't happening); noise looks like both numbers being a
  few MB higher than expected but the *relative* gap still holding.
- 6.12 depends on exactly two TTL expiries landing inside a 150s window at
  1rps against a 60s TTL. A genuine failure is zero spikes in the degraded
  run; noise is a spike count off by one in either direction from a slow
  request unrelated to cache expiry. Re-run before concluding the mechanism
  is broken.

**`test_clean_baseline_is_healthy` (G7), `test_nightmare_degrades_but_survives`
(G8), and `test_blocking_event_loop_serializes_concurrent_requests`'s baseline
assertion (6.7)** are sensitive to host load discovered empirically during
Phase 0b implementation, not just theorized:

- All three compare a "fast path" latency (mostly cache hits, or a single
  request) against a "slow path" one measured moments later, under real
  concurrent load on whatever machine is running the suite. The gap between
  them is genuine, but its exact ratio swings with momentary CPU/scheduler
  contention -- a background process, a thermal-throttled laptop, or simply
  another test having just finished can shift a run from comfortably passing
  to just short.
- This is the same risk `test_clean_baseline_is_healthy`'s own docstring
  already names ("if this fails on a slow machine, that machine cannot run
  the pathology suite meaningfully either") -- empirical testing during
  Phase 0b confirmed it also reaches G8 and 6.7's concurrency-ratio
  assertions specifically, at roughly a 60-80% pass rate on ordinary
  development hardware (one 2024-era laptop, Docker Desktop/OrbStack, no
  other heavy load). Re-run before concluding a genuine regression; a
  consistent, repeatable failure (not just an occasional miss) is the signal
  worth investigating.
- `test_pool_starvation_plateaus_throughput` (6.5) was **removed** during
  Phase 0b for a related but distinct reason: it wasn't flaky, it failed
  consistently, on its baseline (no chaos flags) assertion that throughput
  scales >= 2.5x from concurrency=5 to concurrency=25. Isolated reproduction
  (a bare `GET /healthz` with zero backend work, measured from inside the
  gateway container, bypassing Docker's networking entirely) showed the same
  flat-to-negative scaling regardless of any chaos flag or connection pool
  size. Root cause: `phase-0-base-project-spec.md` §3 mandates uvicorn
  running a single worker per service -- there is no second process or core
  for one service to spread load across, so throughput cannot scale with
  added concurrency in this deployment, independent of pool_starvation. The
  pool_starvation pathology itself (§7) is still implemented in full
  (`services/ledger/db.py`'s dual-engine pool selection); only this one
  differential test, whose premise conflicted with the spec's own mandated
  architecture, was removed.

## Seed determinism

G4 requires "re-running the seed with the same `SEED` yields an identical
manifest." A full re-seed takes up to 60s (per the base spec's own budget),
which would blow the `seed` marker's <20s budget and the combined
`test-contract` <60s budget on every single run, and it's destructive to
whatever other tests' fixture data is live at the time. `test_seed_data.py`
therefore only asserts that the manifest records the `SEED` value it was
generated with (`test_manifest_records_the_configured_seed`) as a fast proxy.
**Verify true determinism manually, occasionally, not as part of CI:**

```bash
make reset && cp db/seed_manifest.json /tmp/manifest-1.json
make reset && diff /tmp/manifest-1.json db/seed_manifest.json
```

## Log-counting migration (Phase 3/4)

6.8, 6.9, and 6.14 count `docker compose logs` lines via
`harness/containers.py::count_log_lines` -- deliberately the only place that
shells out for this, so there's exactly one place to change. From **Phase 3**,
rewrite these against structured logs via LogQL. From **Phase 4**, rewrite
them again against span counts (a TraceQL query for client spans within a
trace answers "how many lookups did this transaction make?" directly). Keep
all three generations' diffs if you can -- they're the clearest demonstration
of what each signal is for, on a problem you'll already understand cold.

## `test_no_observability.py` is deleted at Phase 2

It guards the Phase 0 "before" state (no OTel, no `prometheus_client`, no
structured logging, no `/metrics`). Once instrumentation exists, every
assertion in it becomes actively wrong on purpose. Delete the file; don't
maintain it.

## Goal-to-test traceability

| Goal | Description | Test file(s) |
|---|---|---|
| G1 | End-to-end transaction | `test_contract_transactions.py`, `test_contract_insights.py` |
| G2 | Really seven services | `test_contract_health.py` |
| G3 | Async work, two consumer groups | `test_contract_async.py` |
| G4 | Seed data integrity | `test_seed_data.py` |
| G5 | Chaos control, live reconfig | `test_contract_chaos.py` |
| G6 | Per-pathology differential (§6.1-6.14) | `test_pathologies.py` |
| G7 | Clean baseline is healthy | `test_baseline_health.py` |
| G8 | Nightmare degrades, doesn't collapse | `test_nightmare.py` |
| G9 | Failure containment | `test_contract_health.py` (sanctions stop/timeout paths) |
| G10 | No observability yet | `test_no_observability.py` (deleted at Phase 2) |
| G11 | Frontend served | `test_contract_frontend.py` |

Every test's docstring names its goal ID (and, for G6, its §6.x item number).