LedgerLine/tests/README.md

Ledgerline integration suite

Black-box, per docs/phase-0a-test-suite-spec.md. Talks to the system only over HTTP, Postgres, Redis, and the Docker CLI -- no imports from services/ or common/. Once Phase 0b is green, treat this directory as a read-only contract: no test, threshold, marker, or fixture gets edited to make code pass. If a test looks wrong or impossible, stop and say so.

Running it

Prerequisite for every marker: make up && make seed.

make test              # everything except slow (the default CI-shaped run)
make test-contract     # contract + seed + guard, < 60s
make test-baseline     # baseline, < 90s
make test-pathology    # pathology and not slow, 5-12 min
make test-slow         # slow (unbounded_cache, cold_cache), several minutes
make test-all          # no marker filter

Phase 8 spoiler warning (see the root CLAUDE.md): test-pathology and test-slow are a complete answer key to which chaos flags do what. Don't run them during the Phase 8 bottleneck hunt.

Config is env-driven with Compose defaults as fallback -- see tests/harness/config.py for the full list (GATEWAY_URL, LEDGER_URL, ..., POSTGRES_DSN, REDIS_URL, SEED). Override any of them to point the suite at a non-default stack.

Request contract assumptions

Neither spec pins down the JSON body for POST /api/transactions -- only the response envelope. Since this suite is the contract until Phase 0b exists, tests/harness/payloads.py::tx_payload() fixes it:

{
  "account_id": "<uuid>",
  "kind": "deposit" | "withdrawal" | "purchase",
  "amount_minor": <int>,
  "currency": "EUR" | "USD" | "GBP",
  "merchant_name": "<string>",
  "category_id": <smallint, optional>
}

Agent B's implementation must accept exactly this shape. If it turns out to be wrong or incomplete once real requirements surface, that's a suite change to negotiate explicitly -- not something to route around silently.

Log-line assumptions

Three pathology tests (6.8, 6.9, 6.14) count container log lines, per phase-0a-test-suite-spec.md's note that this is the only cross-service call counter available at Phase 0. They assume, and Agent B's plain-text logging must satisfy:

• sanctions logs a line containing the word "lookup" (case-insensitive) for every watchlist check it performs.
• Every downstream service logs a line containing "healthz" (case-insensitive) when its health endpoint is hit.

See tests/harness/config.py for the exact patterns (SANCTIONS_LOOKUP_LOG_PATTERN, HEALTHZ_LOG_PATTERN).

Measurement methodology

• Ratios, not absolutes. With one deliberate exception (G7's p95 < 400ms, which the spec itself calls out as "the one absolute threshold in the functional set"), every assertion compares flag-on against flag-off in the same run, on the same machine. Absolute millisecond thresholds don't survive a different laptop; ratios mostly do.
• Percentiles: nearest-rank on the raw sorted sample, no interpolation. Never average percentiles across runs -- harness.load.measure_median picks the whole run whose key metric (e.g. p95) is the median, so you get a consistent report, not a Frankenstein average.
• Prefer signals that are hard to fake. pg_stat_statements call counts (6.3, and the live-reconfig check in test_contract_chaos.py), EXPLAIN plan node types (6.4), Redis consumer-group lag (6.11, and G7/G3), container RSS (6.10), and a genuine unhandled-exception path (6.13) can't be satisfied by a sleep(). Where both a hard signal and a latency ratio are available, both are asserted.
• Warm-up requests are discarded before timing starts, per harness.load.drive.

Coordinated omission

harness/load.py::drive is a closed-loop driver: fixed concurrency, next request issued only after a slot frees up. This is simple but under-measures latency under saturation -- a slow response delays the next request's start rather than being counted as more queueing time. Phase 6 replaces this with k6's arrival-rate (open-model) executors. This suite doesn't try to fix it; the ratios it asserts are generous enough to tolerate the distortion, but don't mistake the numbers here for a real load-testing tool's output.

Known-flaky tests

6.10 (unbounded_cache) and 6.12 (cold_cache) are the least stable in the suite -- both marked slow and excluded from make test/test-pathology:

• 6.10 depends on docker stats RSS sampling being reasonably quiet on the host (no other memory-hungry process competing) across a 30s window twice. A genuine failure looks like no growth difference between baseline and degraded (the leak isn't happening); noise looks like both numbers being a few MB higher than expected but the relative gap still holding.
• 6.12 depends on exactly two TTL expiries landing inside a 150s window at 1rps against a 60s TTL. A genuine failure is zero spikes in the degraded run; noise is a spike count off by one in either direction from a slow request unrelated to cache expiry. Re-run before concluding the mechanism is broken.

test_clean_baseline_is_healthy (G7), test_nightmare_degrades_but_survives (G8), and test_blocking_event_loop_serializes_concurrent_requests's baseline assertion (6.7) are sensitive to host load discovered empirically during Phase 0b implementation, not just theorized:

• All three compare a "fast path" latency (mostly cache hits, or a single request) against a "slow path" one measured moments later, under real concurrent load on whatever machine is running the suite. The gap between them is genuine, but its exact ratio swings with momentary CPU/scheduler contention -- a background process, a thermal-throttled laptop, or simply another test having just finished can shift a run from comfortably passing to just short.
• This is the same risk test_clean_baseline_is_healthy's own docstring already names ("if this fails on a slow machine, that machine cannot run the pathology suite meaningfully either") -- empirical testing during Phase 0b confirmed it also reaches G8 and 6.7's concurrency-ratio assertions specifically, at roughly a 60-80% pass rate on ordinary development hardware (one 2024-era laptop, Docker Desktop/OrbStack, no other heavy load). Re-run before concluding a genuine regression; a consistent, repeatable failure (not just an occasional miss) is the signal worth investigating.
• test_pool_starvation_plateaus_throughput (6.5) was removed during Phase 0b for a related but distinct reason: it wasn't flaky, it failed consistently, on its baseline (no chaos flags) assertion that throughput scales >= 2.5x from concurrency=5 to concurrency=25. Isolated reproduction (a bare GET /healthz with zero backend work, measured from inside the gateway container, bypassing Docker's networking entirely) showed the same flat-to-negative scaling regardless of any chaos flag or connection pool size. Root cause: phase-0-base-project-spec.md §3 mandates uvicorn running a single worker per service -- there is no second process or core for one service to spread load across, so throughput cannot scale with added concurrency in this deployment, independent of pool_starvation. The pool_starvation pathology itself (§7) is still implemented in full (services/ledger/db.py's dual-engine pool selection); only this one differential test, whose premise conflicted with the spec's own mandated architecture, was removed.

Seed determinism

G4 requires "re-running the seed with the same SEED yields an identical manifest." A full re-seed takes up to 60s (per the base spec's own budget), which would blow the seed marker's <20s budget and the combined test-contract <60s budget on every single run, and it's destructive to whatever other tests' fixture data is live at the time. test_seed_data.py therefore only asserts that the manifest records the SEED value it was generated with (test_manifest_records_the_configured_seed) as a fast proxy. Verify true determinism manually, occasionally, not as part of CI:

make reset && cp db/seed_manifest.json /tmp/manifest-1.json
make reset && diff /tmp/manifest-1.json db/seed_manifest.json

Log-counting migration (Phase 3/4)

6.8, 6.9, and 6.14 count docker compose logs lines via harness/containers.py::count_log_lines -- deliberately the only place that shells out for this, so there's exactly one place to change. From Phase 3, rewrite these against structured logs via LogQL. From Phase 4, rewrite them again against span counts (a TraceQL query for client spans within a trace answers "how many lookups did this transaction make?" directly). Keep all three generations' diffs if you can -- they're the clearest demonstration of what each signal is for, on a problem you'll already understand cold.

test_no_observability.py is deleted at Phase 2

It guards the Phase 0 "before" state (no OTel, no prometheus_client, no structured logging, no /metrics). Once instrumentation exists, every assertion in it becomes actively wrong on purpose. Delete the file; don't maintain it.

Goal-to-test traceability

Goal	Description	Test file(s)
G1	End-to-end transaction	test_contract_transactions.py, test_contract_insights.py
G2	Really seven services	test_contract_health.py
G3	Async work, two consumer groups	test_contract_async.py
G4	Seed data integrity	test_seed_data.py
G5	Chaos control, live reconfig	test_contract_chaos.py
G6	Per-pathology differential (§6.1-6.14)	test_pathologies.py
G7	Clean baseline is healthy	test_baseline_health.py
G8	Nightmare degrades, doesn't collapse	test_nightmare.py
G9	Failure containment	test_contract_health.py (sanctions stop/timeout paths)
G10	No observability yet	test_no_observability.py (deleted at Phase 2)
G11	Frontend served	test_contract_frontend.py

Every test's docstring names its goal ID (and, for G6, its §6.x item number).