LedgerLine/docs/phase-0a-test-suite-spec.md

Phase 0a — Technical Spec: Ledgerline integration test suite

How to use this file: paste it into the test-authoring agent together with phase-0-base-project-spec.md. That agent writes only tests/. A second agent then implements the services against the resulting suite. See §1 for the workflow contract between them.

---

1. The two-agent workflow

Agent A — test author (this spec)

• Reads phase-0-base-project-spec.md (the system contract) and this file.
• Creates only these paths:

  tests/**            pytest.ini (or pyproject [tool.pytest])
  requirements-test.txt
  tests/README.md
  tests/RED-BASELINE.md
  

  Plus the test-* targets appended to the Makefile.
• Creates nothing under services/, common/, db/, scripts/, observability/ or loadtest/. Not even a stub. If a test needs a fixture from the system, it asserts against the contract in §4 rather than importing application code.
• Imports nothing from the application. Tests talk to the system over HTTP, Postgres, Redis, and the Docker CLI. No from services.gateway import …. This is what keeps them behavioural rather than implementation-coupled.
• Runs the suite against an empty repo and confirms it is red, then writes tests/RED-BASELINE.md listing, per test, why it currently fails (connection refused / 404 / assertion / missing table). A test that passes against nothing is a broken test — find it and fix it.

Agent B — implementer

• Reads phase-0-base-project-spec.md and treats tests/ as a read-only contract.
• Must not modify, delete, skip, xfail, or loosen any test, or change pytest.ini, markers, thresholds, or fixtures. If a test appears wrong, impossible, or contradictory: stop and report it, don't edit it.
• Must implement each pathology by the mechanism described in phase-0-base-project-spec.md §7 — not as a generic sleep() tuned to make a timing assertion pass. Several assertions in §6 are deliberately built so that faking is harder than implementing (query counts, EXPLAIN plan shapes, Redis consumer lag, process RSS). Treat that as the intent everywhere, including where the assertion happens to be simpler.
• Done when make test is green three consecutive runs.

---

2. Design principles for this suite

These are the rules the tests must obey. They matter more than any individual test.

1. Assert goals, not implementation. No test may reference a function name, module path, class, SQL string, log format, or internal attribute. A test may reference the public contract: HTTP routes, status codes, JSON field names, table names, Redis stream and consumer-group names, chaos flag names.
2. Prefer assertions that are hard to satisfy without the real mechanism. Wall-clock latency alone is fakeable with sleep(). Query counts from pg_stat_statements, EXPLAIN plan node types, Redis consumer-group lag, container RSS, and log-line counts on a different service are not. Where both are available, assert on both.
3. Measure differentially, never absolutely. With one exception (§5.7), every pathology test compares flag-on against flag-off in the same run, on the same machine, and asserts a ratio. Absolute millisecond thresholds do not survive a different laptop.
4. Design for a noisy laptop. Generous margins, warm-up requests discarded, median-of-N measurement runs. A test that fails when Slack is open is worse than no test.
5. Fast tests and slow tests are separate. make test-contract must run in under 60 seconds. Pathology tests may take minutes and live behind their own marker.
6. Every test restores state. A test that enables a chaos flag must reset the profile in teardown, even on failure.

---

3. Stack, layout, and execution

Tests run on the host, against the Compose stack on localhost. They do not run inside a container and do not use testcontainers — the stack is started by make up beforehand.

tests/
├─ README.md
├─ RED-BASELINE.md
├─ conftest.py                # fixtures: clients, chaos control, seed manifest
├─ harness/
│  ├─ load.py                 # concurrent request driver + LatencyReport
│  ├─ pg.py                   # pg_stat_statements deltas, EXPLAIN helpers
│  ├─ streams.py              # Redis XINFO GROUPS helpers
│  └─ containers.py           # docker stats / docker logs helpers
├─ test_contract_health.py
├─ test_contract_transactions.py
├─ test_contract_insights.py
├─ test_contract_async.py
├─ test_contract_chaos.py
├─ test_contract_frontend.py
├─ test_seed_data.py
├─ test_baseline_health.py
├─ test_pathologies.py
├─ test_nightmare.py
└─ test_no_observability.py

Dependencies (requirements-test.txt): pytest, pytest-asyncio, pytest-timeout, httpx, psycopg[binary], redis. Docker access via subprocess calling the docker CLI — no SDK.

Markers

Marker	Runtime	Contents
contract	< 60 s	Functional behaviour, chaos-clean
seed	< 20 s	Seed data integrity
baseline	< 90 s	The clean system is genuinely healthy
pathology	5–12 min	One test per pathology, differential
slow	—	Additional marker on the soak-ish ones (unbounded_cache, cold_cache)
guard	< 5 s	The "no observability yet" assertions

Configure pytest.ini with --strict-markers, asyncio_mode = auto, and a global timeout = 300.

Make targets to append

Target	Command
test	everything except slow
test-contract	-m "contract or seed or guard"
test-baseline	-m baseline
test-pathology	-m "pathology and not slow"
test-slow	-m slow
test-all	no marker filter

---

4. Contract additions required for testability

These extend phase-0-base-project-spec.md. Agent B must implement them; Agent A may assume them.

4.1 Seed manifest

scripts/seed.py writes db/seed_manifest.json and a seed_manifest table (single row, data jsonb) with the same content. Tests read the table, so they don't depend on a host file path.

{
  "seed": 42,
  "generated_at": "…",
  "counts": { "accounts": 800, "transactions": 250000,
              "merchant_categories": 45, "sanctions_entries": 12000,
              "malformed_transactions": 5013 },
  "hot_account_ids": ["…", "…", "…", "…", "…"],
  "cold_account_ids_sample": ["…"],            // 50 non-hot accounts, for spread tests
  "sanctioned_account_ids": ["…"],             // accounts whose holder matches the watchlist
  "poison_account_ids": ["…"],                 // accounts owning malformed rows
  "clean_account_id": "…"                      // no malformed rows, not hot, not sanctioned
}

4.2 Response envelope

POST /api/transactions → 201 on success:

{ "transaction_id": "<uuid>", "status": "settled|rejected",
  "risk_score": 0.0-1.0, "balance_minor": 0,
  "summary": { }, "request_id": "<uuid>", "elapsed_ms": 0 }

Errors return a consistent envelope with the appropriate status:

{ "error": "<machine_readable_code>", "detail": "<human string>",
  "request_id": "<uuid>" }

Status mapping: 400 invalid input, 404 unknown account, 422 validation, 502 downstream returned an error, 504 downstream timed out, 500 unhandled — which is exactly what poison_data produces, and what §6.12 asserts.

4.3 Chaos API

• GET /api/admin/chaos → {"profile": "<name|custom>", "flags": {"<flag>": bool, …}} containing all thirteen flags, always.
• POST /api/admin/chaos → {"flag": "<name>", "enabled": bool}; returns the full new state. Unknown flag → 400.
• POST /api/admin/chaos/profile → {"profile": "clean|light|nightmare|random"}; returns the full new state. For random, the response reports the flags — but make chaos-random must not print them, and the seed is never logged.
• GET /api/health/all → {"services": {"<name>": "ok"|"unhealthy", …}} for all six HTTP services.

4.4 pg_stat_statements

The Postgres image must load pg_stat_statements (shared_preload_libraries) and the extension must be created by db/schema.sql. Tests use pg_stat_statements_reset() around a single request to count queries.

4.5 Deterministic seeding

SEED env var, default 42. Same seed ⇒ identical manifest, identical row counts, identical hot/poison account IDs.

---

5. Goal-to-test traceability

Every test must map to one of these goals. Put the goal ID in the test's docstring. This is what keeps the suite about intent rather than mechanics.

ID	Goal	Where
G1	A transaction traverses all seven services and returns a coherent result	§5.1
G2	The system is genuinely multi-service, not a collapsed monolith	§5.2
G3	Async work happens after the response, via two independent consumer groups	§5.3
G4	The seed data is large, skewed, and contains the intended poison rows	§5.4
G5	Chaos flags toggle at runtime without a restart	§5.5
G6	Each pathology measurably degrades the system when enabled	§6
G7	The clean baseline is actually healthy — otherwise G6 means nothing	§5.6
G8	Nightmare mode degrades but does not collapse	§5.7
G9	Failure is contained: timeouts fire, the gateway doesn't wedge	§5.8
G10	No observability tooling is present yet	§5.9
G11	The frontend and its supporting endpoints are served	§5.10

5.1 G1 — end-to-end transaction (contract)

• Submitting a valid transaction for clean_account_id returns 201 with the full envelope; transaction_id is a UUID; status ∈ {settled, rejected}; risk_score ∈ [0,1]; elapsed_ms > 0.
• The transaction is durable: a row with that ID exists in transactions with a terminal status within 2 s.
• The balance moves in the correct direction for a deposit and for a withdrawal, by exactly amount_minor, when the transaction settles.
• A transaction for a sanctioned_account_ids holder yields a materially higher risk_score than one for clean_account_id (assert the ordering, not a value).
• GET /api/accounts/{id}/insights returns a summary for a seeded account.
• Unknown account → 404 with the error envelope. Negative amount on a deposit → 400 or 422. Missing required field → 422.

5.2 G2 — really seven services (contract)

• All six HTTP services answer /healthz on their own port with their own name.
• /readyz returns 200 for all six.
• GET /api/health/all reports all six as ok.
• Stopping sanctions (docker compose stop sanctions) causes GET /api/health/all to report it unhealthy within 15 s, and POST /api/transactions to fail with 502 or 504 — not 200. Restart it in teardown and wait for ready. (This is the test that catches a collapsed monolith: if transactions still succeed with sanctions stopped, the service isn't really in the path.)

5.3 G3 — asynchronous work (contract)

• After a successful transaction, a row appears in notifications and in audit_events for that transaction_id within 10 s.
• Both appear — proving two independent consumer groups, not one consumer doing both jobs.
• XINFO GROUPS stream:transactions reports exactly two groups, named notifier and audit.
• The API response returns before the notification row exists at least once across 20 attempts, proving the work is genuinely off the request path. (Sample the DB immediately on response; assert at least one miss. Tolerant of a fast machine winning the race sometimes — hence 20 attempts, not 1.)

5.4 G4 — seed data (seed)

Against the manifest and the database:

• Row counts match the manifest and the spec's targets within ±2%.
• Exactly 5 accounts have is_hot = true, and they match hot_account_ids.
• The 5 hot accounts own between 20% and 40% of all transactions.
• Transactions span ≥ 110 distinct days.
• Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each).
• Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points).
• Malformed rows exist in all five described varieties: null currency, negative amount on a deposit, non-ASCII merchant name, future created_at, orphan category_id. Assert at least one of each, and 1%–4% overall.
• ≥ 30 sanctions entries match seeded holder names.
• Re-running the seed with the same SEED yields an identical manifest (compare the counts and hot_account_ids).
• Neither transactions(account_id, created_at) nor transactions(category_id) exists in pg_indexes after a fresh seed.

5.5 G5 — chaos control (contract)

• GET /api/admin/chaos lists all thirteen flags by name.
• Toggling a single flag changes only that flag.
• Each profile sets the expected shape: clean → all false; nightmare → all true; light → exactly the two documented flags.
• No restart: with chaos-clean set, record the insights latency; enable n_plus_one via the API; without restarting any container, the query count for the same request rises. Assert on the query-count delta (§6.3's helper), so this test proves live reconfiguration rather than timing noise.
• Unknown flag name → 400.
• Teardown restores clean.

5.6 G7 — the clean baseline is healthy (baseline)

With chaos-clean, after 20 discarded warm-up requests, drive 200 requests at concurrency 10 against random cold accounts:

• error rate == 0
• p95 < 400 ms (the one absolute threshold in the functional set; if it fails on a slow machine, that machine cannot run the pathology suite meaningfully either, and the failure message must say so)
• p99 / p50 < 5 — the clean system has no fat tail
• insights for a cold account issues fewer than 10 SQL statements
• Redis consumer lag returns to 0 within 15 s of the burst finishing

5.7 G8 — nightmare degrades but survives (pathology)

Same driver, chaos-clean then chaos-nightmare, 300 requests at concurrency 20 each:

• nightmare p95 ≥ 5× clean p95
• nightmare error rate between 2% and 15% — degraded, not dead
• at least 80% of requests still return a valid envelope (either success or a structured error, never a connection reset or an empty body)
• GET /api/health/all still responds within 5 s during the burst

5.8 G9 — failure containment (contract)

• With sanctions_errors at its maximum and retry_storm off, no request takes longer than 10 s. The 3 s downstream timeout must actually fire.
• With sanctions stopped entirely, POST /api/transactions returns 502/504 within 10 s, and the gateway still answers /healthz in under 1 s throughout.
• No request ever returns a 200 with a null transaction_id.

5.9 G10 — no observability yet (guard)

Static and runtime assertions that the Phase 0 "before" state is intact:

• No file under services/ or common/ imports opentelemetry, prometheus_client, structlog, sentry_sdk, or elasticapm.
• Neither requirements*.txt nor pyproject.toml lists them (excluding requirements-test.txt).
• No service exposes /metrics (assert 404 on all six).
• docker compose logs gateway emits plain text, not JSON: sample 20 lines and assert none parse as a JSON object.

This test class is expected to be deleted at the start of Phase 2, not maintained. Say so in its module docstring.

5.10 G11 — frontend (contract)

• GET / returns 200, text/html, and references app.js and style.css.
• Those two assets return 200 with sensible content types.
• GET /api/accounts returns ≥ 100 accounts, each with id, holder_name, currency, balance_minor.

No browser automation. Serving and shape only.

---

6. G6 — pathology tests

One test per flag. Each follows the same skeleton:

1. set chaos-clean
2. warm up (discard)
3. measure  →  baseline
4. enable exactly one flag via the API
5. warm up (discard)
6. measure  →  degraded
7. assert the ratio, with the failure message printing both measurements
8. teardown: chaos-clean

Measurement runs the median of 3 for latency-based assertions. Failure messages must always include both raw numbers — a bare assert ratio > 3 is useless at 2 a.m.

§	Flag	Primary signal (hard to fake)	Secondary signal	Assertion
6.1	slow_sanctions	—	p99/p50 of POST /api/transactions	degraded p99/p50 ≥ 6, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99
6.2	sanctions_errors	count of 502/504 responses over 300 requests	—	degraded error rate ∈ [1%, 10%]; baseline == 0
6.3	n_plus_one	pg_stat_statements call delta across one insights request	latency ratio	degraded ≥ 50 statements; baseline ≤ 10; latency ratio ≥ 2
6.4	missing index (make drop-index / fix-index, not a chaos flag)	EXPLAIN (ANALYZE, FORMAT JSON) on the insights aggregation contains a Seq Scan on transactions; after fix-index it contains an Index Scan/Bitmap Index Scan	latency ratio	plan node types as stated; unindexed latency ≥ 3× indexed. Teardown restores the dropped state.
6.5	pool_starvation	throughput (req/s) at concurrency 5 vs 25	p95 ratio	with the flag on, throughput at c=25 is ≤ 1.4× throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25
6.6	row_lock_contention	25 concurrent transactions against one hot account vs 25 spread across 25 cold accounts, both with the flag on	—	single-hot-account p95 ≥ 3× spread p95; with the flag off the same comparison is ≤ 1.8×
6.7	blocking_event_loop	wall-clock for 10 concurrent requests ÷ wall-clock for 1 request	—	degraded ratio ≥ 6 (serialised); baseline ratio ≤ 2.5 (concurrent)
6.8	retry_storm	count of sanctions container log lines matching a lookup, per transaction, with sanctions_errors also on	—	degraded ≥ 2.0 lookups/transaction; baseline (retry off, errors on) ≤ 1.2. Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.
6.9	no_batching	sanctions lookup log lines per transaction, sanctions_errors off	latency ratio	degraded ≥ 2.0 lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5
6.10	unbounded_cache (slow)	insights container RSS via docker stats before vs after 2000 requests with distinct query parameters, sampled 30 s after the burst ends	—	degraded RSS growth ≥ 20 MB and still elevated at +30 s; baseline growth ≤ 8 MB
6.11	slow_consumer	XINFO GROUPS stream:transactions → lag/pending for the notifier group after a 150-transaction burst	drain time	degraded lag ≥ 30 at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s
6.12	cold_cache (slow)	same insights request at 1 rps for 150 s; count samples > 3× the median	—	degraded ≥ 2 spikes (a 60 s TTL over 150 s); baseline ≤ 1
6.13	poison_data	request against each poison_account_ids entry	—	degraded produces ≥ 1 response with status 500; baseline produces zero 500s across the same requests (4xx or 200 only)
6.14	chatty_health	/healthz hits on downstream containers during a single transaction, counted from logs	—	degraded ≥ 5 healthz lines across downstreams per transaction; baseline == 0

Note on log-line counting (6.8, 6.9, 6.14). At Phase 0 the plain-text log is the only cross-service observability surface that exists — which is precisely the deficiency this whole project sets out to fix. Implement these helpers behind harness/containers.py::count_log_lines(service, pattern, since) so there is exactly one place to change. From Phase 3 these assertions should be rewritten against structured logs, and from Phase 4 against span counts — at which point they become both simpler and more robust. Note this in tests/README.md; it is a deliberate teaching artefact, not a shortcut.

Anti-fake note for Agent B. 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied by inserting a sleep(): they require, respectively, real repeated queries, a real missing index, real unbounded allocation, real consumer slowness, and a real unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12) are stated as ratios for the same reason — passing them by faking is more work than implementing the mechanism described in the Phase 0 spec.

---

7. The measurement harness

harness/load.py provides one primitive:

async def drive(
    method: str, url_factory: Callable[[], tuple[str, dict]],
    total: int, concurrency: int,
    warmup: int = 20, timeout: float = 15.0,
) -> LatencyReport

LatencyReport carries count, ok, errors, error_rate, status_counts, p50, p95, p99, max, throughput_rps, wall_seconds, plus a __str__ suitable for dropping straight into an assertion message.

Rules:

• Warm-up requests are issued and discarded before timing starts.
• Percentiles from the raw sorted sample, nearest-rank. No interpolation, no averaging of percentiles across runs — median of the p95s, never the mean.
• A closed-loop driver (fixed concurrency) is used deliberately for its simplicity here. Note in tests/README.md that this under-measures latency under saturation (coordinated omission) and that Phase 6 replaces it with k6's arrival-rate executors. Don't try to fix it in the test suite.
• measure_median(n=3, fn) helper for the pathology tests.

harness/pg.py:

@contextmanager
def statement_count(dsn) -> Counter   # resets pg_stat_statements, yields, returns delta
def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON)
def plan_node_types(plan: dict) -> set[str]

harness/streams.py: group_lag(group), wait_for_drain(group, timeout).

harness/containers.py: rss_mb(service), count_log_lines(service, pattern, since), stop(service), start(service), wait_ready(service, timeout).

---

8. Fixtures (conftest.py)

Fixture	Scope	Purpose
stack_up	session	Assert the Compose stack is running and all six services are ready; fail fast with a clear message ("run make up && make seed first") rather than timing out
seeded	session	Assert the seed manifest exists and counts are non-zero; fail fast otherwise
manifest	session	Parsed seed manifest
api	function	httpx.AsyncClient against the gateway, 15 s timeout
dsn	session	Postgres DSN from env
redis_client	session	—
chaos	function	chaos.set_profile(name), chaos.set_flag(name, bool), chaos.get(). Autouse teardown resets to clean, even on failure
fresh_balances	function	Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs

Every environment value (ports, DSN, Redis URL) comes from env with the Compose defaults as fallbacks. No hardcoded values buried in test bodies.

---

9. tests/README.md — what Agent A must write

• How to run each marker set, and the prerequisite (make up && make seed).
• The measurement methodology and why ratios are used rather than absolutes.
• The known-flaky list: 6.10 and 6.12 are the least stable; document the retry strategy and what a genuine failure looks like versus noise.
• The coordinated-omission caveat on the closed-loop driver.
• The Phase 3/4 migration note for the log-counting helpers.
• An explicit statement that test_no_observability.py is deleted at Phase 2.

---

10. Acceptance criteria

For Agent A:

1. pytest --collect-only succeeds and every test carries exactly one goal ID in its docstring and at least one marker.
2. grep -rn "from services\|from common\|import services\|import common" tests/ returns nothing.
3. Every §5 and §6 item has a corresponding test; tests/README.md contains the goal-to-test table.
4. The suite is red against an empty repo, and tests/RED-BASELINE.md records the failure reason per test. No test passes vacuously.
5. make test-contract completes (failing) in under 60 s.

For Agent B:

1. make test green three consecutive runs.
2. make test-slow green (may be run once).
3. git diff --stat tests/ against Agent A's commit is empty.
4. docs/PATHOLOGIES.md written, and not summarised in the final response.