# Phase 0a — Technical Spec: Ledgerline integration test suite

> **How to use this file:** paste it into the **test-authoring agent** together
> with `phase-0-base-project-spec.md`. That agent writes only `tests/`.
> A second agent then implements the services against the resulting suite.
> See §1 for the workflow contract between them.

---

## 1. The two-agent workflow

### Agent A — test author (this spec)

- Reads `phase-0-base-project-spec.md` (the system contract) and this file.
- Creates **only** these paths:
  ```
  tests/**            pytest.ini (or pyproject [tool.pytest])
  requirements-test.txt
  tests/README.md
  tests/RED-BASELINE.md
  ```
  Plus the `test-*` targets appended to the `Makefile`.
- Creates **nothing** under `services/`, `common/`, `db/`, `scripts/`,
  `observability/` or `loadtest/`. Not even a stub. If a test needs a fixture
  from the system, it asserts against the contract in §4 rather than importing
  application code.
- **Imports nothing from the application.** Tests talk to the system over HTTP,
  Postgres, Redis, and the Docker CLI. No `from services.gateway import …`.
  This is what keeps them behavioural rather than implementation-coupled.
- Runs the suite against an empty repo and confirms it is **red**, then writes
  `tests/RED-BASELINE.md` listing, per test, *why* it currently fails
  (connection refused / 404 / assertion / missing table). A test that passes
  against nothing is a broken test — find it and fix it.

### Agent B — implementer

- Reads `phase-0-base-project-spec.md` and treats `tests/` as a **read-only
  contract**.
- **Must not modify, delete, skip, xfail, or loosen any test**, or change
  `pytest.ini`, markers, thresholds, or fixtures. If a test appears wrong,
  impossible, or contradictory: **stop and report it**, don't edit it.
- Must implement each pathology by the mechanism described in
  `phase-0-base-project-spec.md` §7 — not as a generic `sleep()` tuned to make a
  timing assertion pass. Several assertions in §6 are deliberately built so that
  faking is harder than implementing (query counts, `EXPLAIN` plan shapes, Redis
  consumer lag, process RSS). Treat that as the intent everywhere, including
  where the assertion happens to be simpler.
- Done when `make test` is green three consecutive runs.

---

## 2. Design principles for this suite

These are the rules the tests must obey. They matter more than any individual
test.

1. **Assert goals, not implementation.** No test may reference a function name,
   module path, class, SQL string, log format, or internal attribute. A test may
   reference the *public contract*: HTTP routes, status codes, JSON field names,
   table names, Redis stream and consumer-group names, chaos flag names.
2. **Prefer assertions that are hard to satisfy without the real mechanism.**
   Wall-clock latency alone is fakeable with `sleep()`. Query counts from
   `pg_stat_statements`, `EXPLAIN` plan node types, Redis consumer-group lag,
   container RSS, and log-line counts on a *different service* are not. Where
   both are available, assert on both.
3. **Measure differentially, never absolutely.** With one exception (§5.7),
   every pathology test compares flag-on against flag-off in the same run, on
   the same machine, and asserts a *ratio*. Absolute millisecond thresholds do
   not survive a different laptop.
4. **Design for a noisy laptop.** Generous margins, warm-up requests discarded,
   median-of-N measurement runs. A test that fails when Slack is open is worse
   than no test.
5. **Fast tests and slow tests are separate.** `make test-contract` must run in
   under 60 seconds. Pathology tests may take minutes and live behind their own
   marker.
6. **Every test restores state.** A test that enables a chaos flag must reset
   the profile in teardown, even on failure.

---

## 3. Stack, layout, and execution

Tests run **on the host**, against the Compose stack on `localhost`. They do not
run inside a container and do not use testcontainers — the stack is started by
`make up` beforehand.

```
tests/
├─ README.md
├─ RED-BASELINE.md
├─ conftest.py                # fixtures: clients, chaos control, seed manifest
├─ harness/
│  ├─ load.py                 # concurrent request driver + LatencyReport
│  ├─ pg.py                   # pg_stat_statements deltas, EXPLAIN helpers
│  ├─ streams.py              # Redis XINFO GROUPS helpers
│  └─ containers.py           # docker stats / docker logs helpers
├─ test_contract_health.py
├─ test_contract_transactions.py
├─ test_contract_insights.py
├─ test_contract_async.py
├─ test_contract_chaos.py
├─ test_contract_frontend.py
├─ test_seed_data.py
├─ test_baseline_health.py
├─ test_pathologies.py
├─ test_nightmare.py
└─ test_no_observability.py
```

Dependencies (`requirements-test.txt`): `pytest`, `pytest-asyncio`,
`pytest-timeout`, `httpx`, `psycopg[binary]`, `redis`. Docker access via
`subprocess` calling the `docker` CLI — no SDK.

### Markers

| Marker | Runtime | Contents |
|---|---|---|
| `contract` | < 60 s | Functional behaviour, chaos-clean |
| `seed` | < 20 s | Seed data integrity |
| `baseline` | < 90 s | The clean system is genuinely healthy |
| `pathology` | 5–12 min | One test per pathology, differential |
| `slow` | — | Additional marker on the soak-ish ones (`unbounded_cache`, `cold_cache`) |
| `guard` | < 5 s | The "no observability yet" assertions |

Configure `pytest.ini` with `--strict-markers`, `asyncio_mode = auto`, and a
global `timeout = 300`.

### Make targets to append

| Target | Command |
|---|---|
| `test` | everything except `slow` |
| `test-contract` | `-m "contract or seed or guard"` |
| `test-baseline` | `-m baseline` |
| `test-pathology` | `-m "pathology and not slow"` |
| `test-slow` | `-m slow` |
| `test-all` | no marker filter |

---

## 4. Contract additions required for testability

These extend `phase-0-base-project-spec.md`. Agent B must implement them; Agent A
may assume them.

### 4.1 Seed manifest

`scripts/seed.py` writes `db/seed_manifest.json` **and** a `seed_manifest` table
(single row, `data jsonb`) with the same content. Tests read the table, so they
don't depend on a host file path.

```json
{
  "seed": 42,
  "generated_at": "…",
  "counts": { "accounts": 800, "transactions": 250000,
              "merchant_categories": 45, "sanctions_entries": 12000,
              "malformed_transactions": 5013 },
  "hot_account_ids": ["…", "…", "…", "…", "…"],
  "cold_account_ids_sample": ["…"],            // 50 non-hot accounts, for spread tests
  "sanctioned_account_ids": ["…"],             // accounts whose holder matches the watchlist
  "poison_account_ids": ["…"],                 // accounts owning malformed rows
  "clean_account_id": "…"                      // no malformed rows, not hot, not sanctioned
}
```

### 4.2 Response envelope

`POST /api/transactions` → **201** on success:

```json
{ "transaction_id": "<uuid>", "status": "settled|rejected",
  "risk_score": 0.0-1.0, "balance_minor": 0,
  "summary": { }, "request_id": "<uuid>", "elapsed_ms": 0 }
```

Errors return a consistent envelope with the appropriate status:

```json
{ "error": "<machine_readable_code>", "detail": "<human string>",
  "request_id": "<uuid>" }
```

Status mapping: `400` invalid input, `404` unknown account, `422` validation,
`502` downstream returned an error, `504` downstream timed out, `500`
unhandled — which is exactly what `poison_data` produces, and what §6.12 asserts.

### 4.3 Chaos API

- `GET /api/admin/chaos` → `{"profile": "<name|custom>", "flags": {"<flag>": bool, …}}`
  containing **all thirteen** flags, always.
- `POST /api/admin/chaos` → `{"flag": "<name>", "enabled": bool}`; returns the
  full new state. Unknown flag → `400`.
- `POST /api/admin/chaos/profile` → `{"profile": "clean|light|nightmare|random"}`;
  returns the full new state. For `random`, the response reports the flags —
  **but `make chaos-random` must not print them**, and the seed is never logged.
- `GET /api/health/all` → `{"services": {"<name>": "ok"|"unhealthy", …}}` for all
  six HTTP services.

### 4.4 `pg_stat_statements`

The Postgres image must load `pg_stat_statements` (`shared_preload_libraries`)
and the extension must be created by `db/schema.sql`. Tests use
`pg_stat_statements_reset()` around a single request to count queries.

### 4.5 Deterministic seeding

`SEED` env var, default `42`. Same seed ⇒ identical manifest, identical row
counts, identical hot/poison account IDs.

---

## 5. Goal-to-test traceability

Every test must map to one of these goals. Put the goal ID in the test's
docstring. This is what keeps the suite about intent rather than mechanics.

| ID | Goal | Where |
|---|---|---|
| **G1** | A transaction traverses all seven services and returns a coherent result | §5.1 |
| **G2** | The system is genuinely multi-service, not a collapsed monolith | §5.2 |
| **G3** | Async work happens after the response, via two independent consumer groups | §5.3 |
| **G4** | The seed data is large, skewed, and contains the intended poison rows | §5.4 |
| **G5** | Chaos flags toggle at runtime without a restart | §5.5 |
| **G6** | Each pathology measurably degrades the system when enabled | §6 |
| **G7** | The clean baseline is actually healthy — otherwise G6 means nothing | §5.6 |
| **G8** | Nightmare mode degrades but does not collapse | §5.7 |
| **G9** | Failure is contained: timeouts fire, the gateway doesn't wedge | §5.8 |
| **G10** | No observability tooling is present yet | §5.9 |
| **G11** | The frontend and its supporting endpoints are served | §5.10 |

### 5.1 G1 — end-to-end transaction (`contract`)

- Submitting a valid transaction for `clean_account_id` returns 201 with the full
  envelope; `transaction_id` is a UUID; `status` ∈ {settled, rejected};
  `risk_score` ∈ [0,1]; `elapsed_ms` > 0.
- The transaction is durable: a row with that ID exists in `transactions` with a
  terminal status within 2 s.
- The balance moves in the correct direction for a deposit and for a withdrawal,
  by exactly `amount_minor`, when the transaction settles.
- A transaction for a `sanctioned_account_ids` holder yields a materially higher
  `risk_score` than one for `clean_account_id` (assert the ordering, not a value).
- `GET /api/accounts/{id}/insights` returns a summary for a seeded account.
- Unknown account → 404 with the error envelope. Negative amount on a deposit →
  400 or 422. Missing required field → 422.

### 5.2 G2 — really seven services (`contract`)

- All six HTTP services answer `/healthz` on their own port with their own name.
- `/readyz` returns 200 for all six.
- `GET /api/health/all` reports all six as ok.
- Stopping `sanctions` (`docker compose stop sanctions`) causes
  `GET /api/health/all` to report it unhealthy within 15 s, and
  `POST /api/transactions` to fail with 502 or 504 — **not** 200. Restart it in
  teardown and wait for ready.
  *(This is the test that catches a collapsed monolith: if transactions still
  succeed with `sanctions` stopped, the service isn't really in the path.)*

### 5.3 G3 — asynchronous work (`contract`)

- After a successful transaction, a row appears in `notifications` **and** in
  `audit_events` for that `transaction_id` within 10 s.
- Both appear — proving two independent consumer groups, not one consumer doing
  both jobs.
- `XINFO GROUPS stream:transactions` reports exactly two groups, named
  `notifier` and `audit`.
- The API response returns **before** the notification row exists at least once
  across 20 attempts, proving the work is genuinely off the request path.
  *(Sample the DB immediately on response; assert at least one miss. Tolerant of
  a fast machine winning the race sometimes — hence 20 attempts, not 1.)*

### 5.4 G4 — seed data (`seed`)

Against the manifest and the database:

- Row counts match the manifest and the spec's targets within ±2%.
- Exactly 5 accounts have `is_hot = true`, and they match `hot_account_ids`.
- The 5 hot accounts own between 20% and 40% of all transactions.
- Transactions span ≥ 110 distinct days.
- Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each).
- Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points).
- Malformed rows exist in all five described varieties: null currency, negative
  amount on a deposit, non-ASCII merchant name, future `created_at`, orphan
  `category_id`. Assert **at least one of each**, and 1%–4% overall.
- ≥ 30 sanctions entries match seeded holder names.
- Re-running the seed with the same `SEED` yields an identical manifest
  (compare the `counts` and `hot_account_ids`).
- Neither `transactions(account_id, created_at)` nor `transactions(category_id)`
  exists in `pg_indexes` after a fresh seed.

### 5.5 G5 — chaos control (`contract`)

- `GET /api/admin/chaos` lists all thirteen flags by name.
- Toggling a single flag changes only that flag.
- Each profile sets the expected shape: `clean` → all false; `nightmare` → all
  true; `light` → exactly the two documented flags.
- **No restart:** with `chaos-clean` set, record the `insights` latency; enable
  `n_plus_one` via the API; without restarting any container, the query count for
  the same request rises. Assert on the query-count delta (§6.3's helper), so
  this test proves live reconfiguration rather than timing noise.
- Unknown flag name → 400.
- Teardown restores `clean`.

### 5.6 G7 — the clean baseline is healthy (`baseline`)

With `chaos-clean`, after 20 discarded warm-up requests, drive 200 requests at
concurrency 10 against random cold accounts:

- error rate == 0
- p95 < 400 ms *(the one absolute threshold in the functional set; if it fails on
  a slow machine, that machine cannot run the pathology suite meaningfully
  either, and the failure message must say so)*
- p99 / p50 < 5 — the clean system has no fat tail
- `insights` for a cold account issues fewer than 10 SQL statements
- Redis consumer lag returns to 0 within 15 s of the burst finishing

### 5.7 G8 — nightmare degrades but survives (`pathology`)

Same driver, `chaos-clean` then `chaos-nightmare`, 300 requests at concurrency 20
each:

- nightmare p95 ≥ **5×** clean p95
- nightmare error rate between **2% and 15%** — degraded, not dead
- at least 80% of requests still return a valid envelope (either success or a
  structured error, never a connection reset or an empty body)
- `GET /api/health/all` still responds within 5 s during the burst

### 5.8 G9 — failure containment (`contract`)

- With `sanctions_errors` at its maximum and `retry_storm` off, no request takes
  longer than 10 s. The 3 s downstream timeout must actually fire.
- With `sanctions` stopped entirely, `POST /api/transactions` returns 502/504
  within 10 s, and the gateway still answers `/healthz` in under 1 s throughout.
- No request ever returns a 200 with a `null` `transaction_id`.

### 5.9 G10 — no observability yet (`guard`)

Static and runtime assertions that the Phase 0 "before" state is intact:

- No file under `services/` or `common/` imports `opentelemetry`,
  `prometheus_client`, `structlog`, `sentry_sdk`, or `elasticapm`.
- Neither `requirements*.txt` nor `pyproject.toml` lists them (excluding
  `requirements-test.txt`).
- No service exposes `/metrics` (assert 404 on all six).
- `docker compose logs gateway` emits plain text, not JSON: sample 20 lines and
  assert none parse as a JSON object.

> This test class is **expected to be deleted at the start of Phase 2**, not
> maintained. Say so in its module docstring.

### 5.10 G11 — frontend (`contract`)

- `GET /` returns 200, `text/html`, and references `app.js` and `style.css`.
- Those two assets return 200 with sensible content types.
- `GET /api/accounts` returns ≥ 100 accounts, each with `id`, `holder_name`,
  `currency`, `balance_minor`.

No browser automation. Serving and shape only.

---

## 6. G6 — pathology tests

**One test per flag.** Each follows the same skeleton:

```
1. set chaos-clean
2. warm up (discard)
3. measure  →  baseline
4. enable exactly one flag via the API
5. warm up (discard)
6. measure  →  degraded
7. assert the ratio, with the failure message printing both measurements
8. teardown: chaos-clean
```

Measurement runs the median of 3 for latency-based assertions. Failure messages
must always include both raw numbers — a bare `assert ratio > 3` is useless at
2 a.m.

| § | Flag | Primary signal (hard to fake) | Secondary signal | Assertion |
|---|---|---|---|---|
| 6.1 | `slow_sanctions` | — | p99/p50 of `POST /api/transactions` | degraded p99/p50 ≥ **6**, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99 |
| 6.2 | `sanctions_errors` | count of 502/504 responses over 300 requests | — | degraded error rate ∈ [1%, 10%]; baseline == 0 |
| 6.3 | `n_plus_one` | `pg_stat_statements` call delta across **one** insights request | latency ratio | degraded ≥ **50** statements; baseline ≤ 10; latency ratio ≥ 2 |
| 6.4 | *missing index* (`make drop-index` / `fix-index`, not a chaos flag) | `EXPLAIN (ANALYZE, FORMAT JSON)` on the insights aggregation contains a `Seq Scan` on `transactions`; after `fix-index` it contains an `Index Scan`/`Bitmap Index Scan` | latency ratio | plan node types as stated; unindexed latency ≥ **3×** indexed. Teardown restores the dropped state. |
| 6.5 | `pool_starvation` | throughput (req/s) at concurrency 5 vs 25 | p95 ratio | with the flag on, throughput at c=25 is ≤ **1.4×** throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25 |
| 6.6 | `row_lock_contention` | 25 concurrent transactions against **one hot account** vs 25 spread across 25 cold accounts, both with the flag on | — | single-hot-account p95 ≥ **3×** spread p95; with the flag off the same comparison is ≤ 1.8× |
| 6.7 | `blocking_event_loop` | wall-clock for 10 concurrent requests ÷ wall-clock for 1 request | — | degraded ratio ≥ **6** (serialised); baseline ratio ≤ 2.5 (concurrent) |
| 6.8 | `retry_storm` | count of `sanctions` container log lines matching a lookup, per transaction, with `sanctions_errors` also on | — | degraded ≥ **2.0** lookups/transaction; baseline (retry off, errors on) ≤ 1.2. *Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.* |
| 6.9 | `no_batching` | `sanctions` lookup log lines per transaction, `sanctions_errors` **off** | latency ratio | degraded ≥ **2.0** lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5 |
| 6.10 | `unbounded_cache` (`slow`) | `insights` container RSS via `docker stats` before vs after 2000 requests with **distinct** query parameters, sampled 30 s after the burst ends | — | degraded RSS growth ≥ **20 MB** and still elevated at +30 s; baseline growth ≤ 8 MB |
| 6.11 | `slow_consumer` | `XINFO GROUPS stream:transactions` → `lag`/`pending` for the `notifier` group after a 150-transaction burst | drain time | degraded lag ≥ **30** at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s |
| 6.12 | `cold_cache` (`slow`) | same insights request at 1 rps for 150 s; count samples > 3× the median | — | degraded ≥ **2** spikes (a 60 s TTL over 150 s); baseline ≤ 1 |
| 6.13 | `poison_data` | request against each `poison_account_ids` entry | — | degraded produces ≥ 1 response with status **500**; baseline produces zero 500s across the same requests (4xx or 200 only) |
| 6.14 | `chatty_health` | `/healthz` hits on downstream containers during a single transaction, counted from logs | — | degraded ≥ **5** healthz lines across downstreams per transaction; baseline == 0 |

**Note on log-line counting (6.8, 6.9, 6.14).** At Phase 0 the plain-text log is
the only cross-service observability surface that exists — which is precisely the
deficiency this whole project sets out to fix. Implement these helpers behind
`harness/containers.py::count_log_lines(service, pattern, since)` so there is
exactly one place to change. **From Phase 3 these assertions should be rewritten
against structured logs, and from Phase 4 against span counts** — at which point
they become both simpler and more robust. Note this in `tests/README.md`; it is a
deliberate teaching artefact, not a shortcut.

**Anti-fake note for Agent B.** 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied
by inserting a `sleep()`: they require, respectively, real repeated queries, a
real missing index, real unbounded allocation, real consumer slowness, and a real
unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12)
are stated as ratios for the same reason — passing them by faking is more work
than implementing the mechanism described in the Phase 0 spec.

---

## 7. The measurement harness

`harness/load.py` provides one primitive:

```python
async def drive(
    method: str, url_factory: Callable[[], tuple[str, dict]],
    total: int, concurrency: int,
    warmup: int = 20, timeout: float = 15.0,
) -> LatencyReport
```

`LatencyReport` carries `count, ok, errors, error_rate, status_counts,
p50, p95, p99, max, throughput_rps, wall_seconds`, plus a `__str__` suitable for
dropping straight into an assertion message.

Rules:

- Warm-up requests are issued and discarded before timing starts.
- Percentiles from the raw sorted sample, nearest-rank. No interpolation, no
  averaging of percentiles across runs — **median of the p95s**, never the mean.
- A closed-loop driver (fixed concurrency) is used deliberately for its
  simplicity here. Note in `tests/README.md` that this under-measures latency
  under saturation (coordinated omission) and that Phase 6 replaces it with k6's
  arrival-rate executors. Don't try to fix it in the test suite.
- `measure_median(n=3, fn)` helper for the pathology tests.

`harness/pg.py`:

```python
@contextmanager
def statement_count(dsn) -> Counter   # resets pg_stat_statements, yields, returns delta
def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON)
def plan_node_types(plan: dict) -> set[str]
```

`harness/streams.py`: `group_lag(group)`, `wait_for_drain(group, timeout)`.

`harness/containers.py`: `rss_mb(service)`, `count_log_lines(service, pattern,
since)`, `stop(service)`, `start(service)`, `wait_ready(service, timeout)`.

---

## 8. Fixtures (`conftest.py`)

| Fixture | Scope | Purpose |
|---|---|---|
| `stack_up` | session | Assert the Compose stack is running and all six services are ready; **fail fast with a clear message** ("run `make up && make seed` first") rather than timing out |
| `seeded` | session | Assert the seed manifest exists and counts are non-zero; fail fast otherwise |
| `manifest` | session | Parsed seed manifest |
| `api` | function | `httpx.AsyncClient` against the gateway, 15 s timeout |
| `dsn` | session | Postgres DSN from env |
| `redis_client` | session | — |
| `chaos` | function | `chaos.set_profile(name)`, `chaos.set_flag(name, bool)`, `chaos.get()`. **Autouse teardown resets to `clean`**, even on failure |
| `fresh_balances` | function | Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs |

Every environment value (ports, DSN, Redis URL) comes from env with the Compose
defaults as fallbacks. No hardcoded values buried in test bodies.

---

## 9. `tests/README.md` — what Agent A must write

- How to run each marker set, and the prerequisite (`make up && make seed`).
- The measurement methodology and why ratios are used rather than absolutes.
- The known-flaky list: 6.10 and 6.12 are the least stable; document the retry
  strategy and what a genuine failure looks like versus noise.
- The coordinated-omission caveat on the closed-loop driver.
- The Phase 3/4 migration note for the log-counting helpers.
- An explicit statement that `test_no_observability.py` is deleted at Phase 2.

---

## 10. Acceptance criteria

For **Agent A**:

1. `pytest --collect-only` succeeds and every test carries exactly one goal ID in
   its docstring and at least one marker.
2. `grep -rn "from services\|from common\|import services\|import common" tests/`
   returns nothing.
3. Every §5 and §6 item has a corresponding test; `tests/README.md` contains the
   goal-to-test table.
4. The suite is **red** against an empty repo, and `tests/RED-BASELINE.md`
   records the failure reason per test. No test passes vacuously.
5. `make test-contract` completes (failing) in under 60 s.

For **Agent B**:

1. `make test` green three consecutive runs.
2. `make test-slow` green (may be run once).
3. `git diff --stat tests/` against Agent A's commit is empty.
4. `docs/PATHOLOGIES.md` written, and not summarised in the final response.