Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
509 lines
24 KiB
Markdown
509 lines
24 KiB
Markdown
# Phase 0a — Technical Spec: Ledgerline integration test suite
|
||
|
||
> **How to use this file:** paste it into the **test-authoring agent** together
|
||
> with `phase-0-base-project-spec.md`. That agent writes only `tests/`.
|
||
> A second agent then implements the services against the resulting suite.
|
||
> See §1 for the workflow contract between them.
|
||
|
||
---
|
||
|
||
## 1. The two-agent workflow
|
||
|
||
### Agent A — test author (this spec)
|
||
|
||
- Reads `phase-0-base-project-spec.md` (the system contract) and this file.
|
||
- Creates **only** these paths:
|
||
```
|
||
tests/** pytest.ini (or pyproject [tool.pytest])
|
||
requirements-test.txt
|
||
tests/README.md
|
||
tests/RED-BASELINE.md
|
||
```
|
||
Plus the `test-*` targets appended to the `Makefile`.
|
||
- Creates **nothing** under `services/`, `common/`, `db/`, `scripts/`,
|
||
`observability/` or `loadtest/`. Not even a stub. If a test needs a fixture
|
||
from the system, it asserts against the contract in §4 rather than importing
|
||
application code.
|
||
- **Imports nothing from the application.** Tests talk to the system over HTTP,
|
||
Postgres, Redis, and the Docker CLI. No `from services.gateway import …`.
|
||
This is what keeps them behavioural rather than implementation-coupled.
|
||
- Runs the suite against an empty repo and confirms it is **red**, then writes
|
||
`tests/RED-BASELINE.md` listing, per test, *why* it currently fails
|
||
(connection refused / 404 / assertion / missing table). A test that passes
|
||
against nothing is a broken test — find it and fix it.
|
||
|
||
### Agent B — implementer
|
||
|
||
- Reads `phase-0-base-project-spec.md` and treats `tests/` as a **read-only
|
||
contract**.
|
||
- **Must not modify, delete, skip, xfail, or loosen any test**, or change
|
||
`pytest.ini`, markers, thresholds, or fixtures. If a test appears wrong,
|
||
impossible, or contradictory: **stop and report it**, don't edit it.
|
||
- Must implement each pathology by the mechanism described in
|
||
`phase-0-base-project-spec.md` §7 — not as a generic `sleep()` tuned to make a
|
||
timing assertion pass. Several assertions in §6 are deliberately built so that
|
||
faking is harder than implementing (query counts, `EXPLAIN` plan shapes, Redis
|
||
consumer lag, process RSS). Treat that as the intent everywhere, including
|
||
where the assertion happens to be simpler.
|
||
- Done when `make test` is green three consecutive runs.
|
||
|
||
---
|
||
|
||
## 2. Design principles for this suite
|
||
|
||
These are the rules the tests must obey. They matter more than any individual
|
||
test.
|
||
|
||
1. **Assert goals, not implementation.** No test may reference a function name,
|
||
module path, class, SQL string, log format, or internal attribute. A test may
|
||
reference the *public contract*: HTTP routes, status codes, JSON field names,
|
||
table names, Redis stream and consumer-group names, chaos flag names.
|
||
2. **Prefer assertions that are hard to satisfy without the real mechanism.**
|
||
Wall-clock latency alone is fakeable with `sleep()`. Query counts from
|
||
`pg_stat_statements`, `EXPLAIN` plan node types, Redis consumer-group lag,
|
||
container RSS, and log-line counts on a *different service* are not. Where
|
||
both are available, assert on both.
|
||
3. **Measure differentially, never absolutely.** With one exception (§5.7),
|
||
every pathology test compares flag-on against flag-off in the same run, on
|
||
the same machine, and asserts a *ratio*. Absolute millisecond thresholds do
|
||
not survive a different laptop.
|
||
4. **Design for a noisy laptop.** Generous margins, warm-up requests discarded,
|
||
median-of-N measurement runs. A test that fails when Slack is open is worse
|
||
than no test.
|
||
5. **Fast tests and slow tests are separate.** `make test-contract` must run in
|
||
under 60 seconds. Pathology tests may take minutes and live behind their own
|
||
marker.
|
||
6. **Every test restores state.** A test that enables a chaos flag must reset
|
||
the profile in teardown, even on failure.
|
||
|
||
---
|
||
|
||
## 3. Stack, layout, and execution
|
||
|
||
Tests run **on the host**, against the Compose stack on `localhost`. They do not
|
||
run inside a container and do not use testcontainers — the stack is started by
|
||
`make up` beforehand.
|
||
|
||
```
|
||
tests/
|
||
├─ README.md
|
||
├─ RED-BASELINE.md
|
||
├─ conftest.py # fixtures: clients, chaos control, seed manifest
|
||
├─ harness/
|
||
│ ├─ load.py # concurrent request driver + LatencyReport
|
||
│ ├─ pg.py # pg_stat_statements deltas, EXPLAIN helpers
|
||
│ ├─ streams.py # Redis XINFO GROUPS helpers
|
||
│ └─ containers.py # docker stats / docker logs helpers
|
||
├─ test_contract_health.py
|
||
├─ test_contract_transactions.py
|
||
├─ test_contract_insights.py
|
||
├─ test_contract_async.py
|
||
├─ test_contract_chaos.py
|
||
├─ test_contract_frontend.py
|
||
├─ test_seed_data.py
|
||
├─ test_baseline_health.py
|
||
├─ test_pathologies.py
|
||
├─ test_nightmare.py
|
||
└─ test_no_observability.py
|
||
```
|
||
|
||
Dependencies (`requirements-test.txt`): `pytest`, `pytest-asyncio`,
|
||
`pytest-timeout`, `httpx`, `psycopg[binary]`, `redis`. Docker access via
|
||
`subprocess` calling the `docker` CLI — no SDK.
|
||
|
||
### Markers
|
||
|
||
| Marker | Runtime | Contents |
|
||
|---|---|---|
|
||
| `contract` | < 60 s | Functional behaviour, chaos-clean |
|
||
| `seed` | < 20 s | Seed data integrity |
|
||
| `baseline` | < 90 s | The clean system is genuinely healthy |
|
||
| `pathology` | 5–12 min | One test per pathology, differential |
|
||
| `slow` | — | Additional marker on the soak-ish ones (`unbounded_cache`, `cold_cache`) |
|
||
| `guard` | < 5 s | The "no observability yet" assertions |
|
||
|
||
Configure `pytest.ini` with `--strict-markers`, `asyncio_mode = auto`, and a
|
||
global `timeout = 300`.
|
||
|
||
### Make targets to append
|
||
|
||
| Target | Command |
|
||
|---|---|
|
||
| `test` | everything except `slow` |
|
||
| `test-contract` | `-m "contract or seed or guard"` |
|
||
| `test-baseline` | `-m baseline` |
|
||
| `test-pathology` | `-m "pathology and not slow"` |
|
||
| `test-slow` | `-m slow` |
|
||
| `test-all` | no marker filter |
|
||
|
||
---
|
||
|
||
## 4. Contract additions required for testability
|
||
|
||
These extend `phase-0-base-project-spec.md`. Agent B must implement them; Agent A
|
||
may assume them.
|
||
|
||
### 4.1 Seed manifest
|
||
|
||
`scripts/seed.py` writes `db/seed_manifest.json` **and** a `seed_manifest` table
|
||
(single row, `data jsonb`) with the same content. Tests read the table, so they
|
||
don't depend on a host file path.
|
||
|
||
```json
|
||
{
|
||
"seed": 42,
|
||
"generated_at": "…",
|
||
"counts": { "accounts": 800, "transactions": 250000,
|
||
"merchant_categories": 45, "sanctions_entries": 12000,
|
||
"malformed_transactions": 5013 },
|
||
"hot_account_ids": ["…", "…", "…", "…", "…"],
|
||
"cold_account_ids_sample": ["…"], // 50 non-hot accounts, for spread tests
|
||
"sanctioned_account_ids": ["…"], // accounts whose holder matches the watchlist
|
||
"poison_account_ids": ["…"], // accounts owning malformed rows
|
||
"clean_account_id": "…" // no malformed rows, not hot, not sanctioned
|
||
}
|
||
```
|
||
|
||
### 4.2 Response envelope
|
||
|
||
`POST /api/transactions` → **201** on success:
|
||
|
||
```json
|
||
{ "transaction_id": "<uuid>", "status": "settled|rejected",
|
||
"risk_score": 0.0-1.0, "balance_minor": 0,
|
||
"summary": { }, "request_id": "<uuid>", "elapsed_ms": 0 }
|
||
```
|
||
|
||
Errors return a consistent envelope with the appropriate status:
|
||
|
||
```json
|
||
{ "error": "<machine_readable_code>", "detail": "<human string>",
|
||
"request_id": "<uuid>" }
|
||
```
|
||
|
||
Status mapping: `400` invalid input, `404` unknown account, `422` validation,
|
||
`502` downstream returned an error, `504` downstream timed out, `500`
|
||
unhandled — which is exactly what `poison_data` produces, and what §6.12 asserts.
|
||
|
||
### 4.3 Chaos API
|
||
|
||
- `GET /api/admin/chaos` → `{"profile": "<name|custom>", "flags": {"<flag>": bool, …}}`
|
||
containing **all thirteen** flags, always.
|
||
- `POST /api/admin/chaos` → `{"flag": "<name>", "enabled": bool}`; returns the
|
||
full new state. Unknown flag → `400`.
|
||
- `POST /api/admin/chaos/profile` → `{"profile": "clean|light|nightmare|random"}`;
|
||
returns the full new state. For `random`, the response reports the flags —
|
||
**but `make chaos-random` must not print them**, and the seed is never logged.
|
||
- `GET /api/health/all` → `{"services": {"<name>": "ok"|"unhealthy", …}}` for all
|
||
six HTTP services.
|
||
|
||
### 4.4 `pg_stat_statements`
|
||
|
||
The Postgres image must load `pg_stat_statements` (`shared_preload_libraries`)
|
||
and the extension must be created by `db/schema.sql`. Tests use
|
||
`pg_stat_statements_reset()` around a single request to count queries.
|
||
|
||
### 4.5 Deterministic seeding
|
||
|
||
`SEED` env var, default `42`. Same seed ⇒ identical manifest, identical row
|
||
counts, identical hot/poison account IDs.
|
||
|
||
---
|
||
|
||
## 5. Goal-to-test traceability
|
||
|
||
Every test must map to one of these goals. Put the goal ID in the test's
|
||
docstring. This is what keeps the suite about intent rather than mechanics.
|
||
|
||
| ID | Goal | Where |
|
||
|---|---|---|
|
||
| **G1** | A transaction traverses all seven services and returns a coherent result | §5.1 |
|
||
| **G2** | The system is genuinely multi-service, not a collapsed monolith | §5.2 |
|
||
| **G3** | Async work happens after the response, via two independent consumer groups | §5.3 |
|
||
| **G4** | The seed data is large, skewed, and contains the intended poison rows | §5.4 |
|
||
| **G5** | Chaos flags toggle at runtime without a restart | §5.5 |
|
||
| **G6** | Each pathology measurably degrades the system when enabled | §6 |
|
||
| **G7** | The clean baseline is actually healthy — otherwise G6 means nothing | §5.6 |
|
||
| **G8** | Nightmare mode degrades but does not collapse | §5.7 |
|
||
| **G9** | Failure is contained: timeouts fire, the gateway doesn't wedge | §5.8 |
|
||
| **G10** | No observability tooling is present yet | §5.9 |
|
||
| **G11** | The frontend and its supporting endpoints are served | §5.10 |
|
||
|
||
### 5.1 G1 — end-to-end transaction (`contract`)
|
||
|
||
- Submitting a valid transaction for `clean_account_id` returns 201 with the full
|
||
envelope; `transaction_id` is a UUID; `status` ∈ {settled, rejected};
|
||
`risk_score` ∈ [0,1]; `elapsed_ms` > 0.
|
||
- The transaction is durable: a row with that ID exists in `transactions` with a
|
||
terminal status within 2 s.
|
||
- The balance moves in the correct direction for a deposit and for a withdrawal,
|
||
by exactly `amount_minor`, when the transaction settles.
|
||
- A transaction for a `sanctioned_account_ids` holder yields a materially higher
|
||
`risk_score` than one for `clean_account_id` (assert the ordering, not a value).
|
||
- `GET /api/accounts/{id}/insights` returns a summary for a seeded account.
|
||
- Unknown account → 404 with the error envelope. Negative amount on a deposit →
|
||
400 or 422. Missing required field → 422.
|
||
|
||
### 5.2 G2 — really seven services (`contract`)
|
||
|
||
- All six HTTP services answer `/healthz` on their own port with their own name.
|
||
- `/readyz` returns 200 for all six.
|
||
- `GET /api/health/all` reports all six as ok.
|
||
- Stopping `sanctions` (`docker compose stop sanctions`) causes
|
||
`GET /api/health/all` to report it unhealthy within 15 s, and
|
||
`POST /api/transactions` to fail with 502 or 504 — **not** 200. Restart it in
|
||
teardown and wait for ready.
|
||
*(This is the test that catches a collapsed monolith: if transactions still
|
||
succeed with `sanctions` stopped, the service isn't really in the path.)*
|
||
|
||
### 5.3 G3 — asynchronous work (`contract`)
|
||
|
||
- After a successful transaction, a row appears in `notifications` **and** in
|
||
`audit_events` for that `transaction_id` within 10 s.
|
||
- Both appear — proving two independent consumer groups, not one consumer doing
|
||
both jobs.
|
||
- `XINFO GROUPS stream:transactions` reports exactly two groups, named
|
||
`notifier` and `audit`.
|
||
- The API response returns **before** the notification row exists at least once
|
||
across 20 attempts, proving the work is genuinely off the request path.
|
||
*(Sample the DB immediately on response; assert at least one miss. Tolerant of
|
||
a fast machine winning the race sometimes — hence 20 attempts, not 1.)*
|
||
|
||
### 5.4 G4 — seed data (`seed`)
|
||
|
||
Against the manifest and the database:
|
||
|
||
- Row counts match the manifest and the spec's targets within ±2%.
|
||
- Exactly 5 accounts have `is_hot = true`, and they match `hot_account_ids`.
|
||
- The 5 hot accounts own between 20% and 40% of all transactions.
|
||
- Transactions span ≥ 110 distinct days.
|
||
- Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each).
|
||
- Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points).
|
||
- Malformed rows exist in all five described varieties: null currency, negative
|
||
amount on a deposit, non-ASCII merchant name, future `created_at`, orphan
|
||
`category_id`. Assert **at least one of each**, and 1%–4% overall.
|
||
- ≥ 30 sanctions entries match seeded holder names.
|
||
- Re-running the seed with the same `SEED` yields an identical manifest
|
||
(compare the `counts` and `hot_account_ids`).
|
||
- Neither `transactions(account_id, created_at)` nor `transactions(category_id)`
|
||
exists in `pg_indexes` after a fresh seed.
|
||
|
||
### 5.5 G5 — chaos control (`contract`)
|
||
|
||
- `GET /api/admin/chaos` lists all thirteen flags by name.
|
||
- Toggling a single flag changes only that flag.
|
||
- Each profile sets the expected shape: `clean` → all false; `nightmare` → all
|
||
true; `light` → exactly the two documented flags.
|
||
- **No restart:** with `chaos-clean` set, record the `insights` latency; enable
|
||
`n_plus_one` via the API; without restarting any container, the query count for
|
||
the same request rises. Assert on the query-count delta (§6.3's helper), so
|
||
this test proves live reconfiguration rather than timing noise.
|
||
- Unknown flag name → 400.
|
||
- Teardown restores `clean`.
|
||
|
||
### 5.6 G7 — the clean baseline is healthy (`baseline`)
|
||
|
||
With `chaos-clean`, after 20 discarded warm-up requests, drive 200 requests at
|
||
concurrency 10 against random cold accounts:
|
||
|
||
- error rate == 0
|
||
- p95 < 400 ms *(the one absolute threshold in the functional set; if it fails on
|
||
a slow machine, that machine cannot run the pathology suite meaningfully
|
||
either, and the failure message must say so)*
|
||
- p99 / p50 < 5 — the clean system has no fat tail
|
||
- `insights` for a cold account issues fewer than 10 SQL statements
|
||
- Redis consumer lag returns to 0 within 15 s of the burst finishing
|
||
|
||
### 5.7 G8 — nightmare degrades but survives (`pathology`)
|
||
|
||
Same driver, `chaos-clean` then `chaos-nightmare`, 300 requests at concurrency 20
|
||
each:
|
||
|
||
- nightmare p95 ≥ **5×** clean p95
|
||
- nightmare error rate between **2% and 15%** — degraded, not dead
|
||
- at least 80% of requests still return a valid envelope (either success or a
|
||
structured error, never a connection reset or an empty body)
|
||
- `GET /api/health/all` still responds within 5 s during the burst
|
||
|
||
### 5.8 G9 — failure containment (`contract`)
|
||
|
||
- With `sanctions_errors` at its maximum and `retry_storm` off, no request takes
|
||
longer than 10 s. The 3 s downstream timeout must actually fire.
|
||
- With `sanctions` stopped entirely, `POST /api/transactions` returns 502/504
|
||
within 10 s, and the gateway still answers `/healthz` in under 1 s throughout.
|
||
- No request ever returns a 200 with a `null` `transaction_id`.
|
||
|
||
### 5.9 G10 — no observability yet (`guard`)
|
||
|
||
Static and runtime assertions that the Phase 0 "before" state is intact:
|
||
|
||
- No file under `services/` or `common/` imports `opentelemetry`,
|
||
`prometheus_client`, `structlog`, `sentry_sdk`, or `elasticapm`.
|
||
- Neither `requirements*.txt` nor `pyproject.toml` lists them (excluding
|
||
`requirements-test.txt`).
|
||
- No service exposes `/metrics` (assert 404 on all six).
|
||
- `docker compose logs gateway` emits plain text, not JSON: sample 20 lines and
|
||
assert none parse as a JSON object.
|
||
|
||
> This test class is **expected to be deleted at the start of Phase 2**, not
|
||
> maintained. Say so in its module docstring.
|
||
|
||
### 5.10 G11 — frontend (`contract`)
|
||
|
||
- `GET /` returns 200, `text/html`, and references `app.js` and `style.css`.
|
||
- Those two assets return 200 with sensible content types.
|
||
- `GET /api/accounts` returns ≥ 100 accounts, each with `id`, `holder_name`,
|
||
`currency`, `balance_minor`.
|
||
|
||
No browser automation. Serving and shape only.
|
||
|
||
---
|
||
|
||
## 6. G6 — pathology tests
|
||
|
||
**One test per flag.** Each follows the same skeleton:
|
||
|
||
```
|
||
1. set chaos-clean
|
||
2. warm up (discard)
|
||
3. measure → baseline
|
||
4. enable exactly one flag via the API
|
||
5. warm up (discard)
|
||
6. measure → degraded
|
||
7. assert the ratio, with the failure message printing both measurements
|
||
8. teardown: chaos-clean
|
||
```
|
||
|
||
Measurement runs the median of 3 for latency-based assertions. Failure messages
|
||
must always include both raw numbers — a bare `assert ratio > 3` is useless at
|
||
2 a.m.
|
||
|
||
| § | Flag | Primary signal (hard to fake) | Secondary signal | Assertion |
|
||
|---|---|---|---|---|
|
||
| 6.1 | `slow_sanctions` | — | p99/p50 of `POST /api/transactions` | degraded p99/p50 ≥ **6**, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99 |
|
||
| 6.2 | `sanctions_errors` | count of 502/504 responses over 300 requests | — | degraded error rate ∈ [1%, 10%]; baseline == 0 |
|
||
| 6.3 | `n_plus_one` | `pg_stat_statements` call delta across **one** insights request | latency ratio | degraded ≥ **50** statements; baseline ≤ 10; latency ratio ≥ 2 |
|
||
| 6.4 | *missing index* (`make drop-index` / `fix-index`, not a chaos flag) | `EXPLAIN (ANALYZE, FORMAT JSON)` on the insights aggregation contains a `Seq Scan` on `transactions`; after `fix-index` it contains an `Index Scan`/`Bitmap Index Scan` | latency ratio | plan node types as stated; unindexed latency ≥ **3×** indexed. Teardown restores the dropped state. |
|
||
| 6.5 | `pool_starvation` | throughput (req/s) at concurrency 5 vs 25 | p95 ratio | with the flag on, throughput at c=25 is ≤ **1.4×** throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25 |
|
||
| 6.6 | `row_lock_contention` | 25 concurrent transactions against **one hot account** vs 25 spread across 25 cold accounts, both with the flag on | — | single-hot-account p95 ≥ **3×** spread p95; with the flag off the same comparison is ≤ 1.8× |
|
||
| 6.7 | `blocking_event_loop` | wall-clock for 10 concurrent requests ÷ wall-clock for 1 request | — | degraded ratio ≥ **6** (serialised); baseline ratio ≤ 2.5 (concurrent) |
|
||
| 6.8 | `retry_storm` | count of `sanctions` container log lines matching a lookup, per transaction, with `sanctions_errors` also on | — | degraded ≥ **2.0** lookups/transaction; baseline (retry off, errors on) ≤ 1.2. *Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.* |
|
||
| 6.9 | `no_batching` | `sanctions` lookup log lines per transaction, `sanctions_errors` **off** | latency ratio | degraded ≥ **2.0** lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5 |
|
||
| 6.10 | `unbounded_cache` (`slow`) | `insights` container RSS via `docker stats` before vs after 2000 requests with **distinct** query parameters, sampled 30 s after the burst ends | — | degraded RSS growth ≥ **20 MB** and still elevated at +30 s; baseline growth ≤ 8 MB |
|
||
| 6.11 | `slow_consumer` | `XINFO GROUPS stream:transactions` → `lag`/`pending` for the `notifier` group after a 150-transaction burst | drain time | degraded lag ≥ **30** at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s |
|
||
| 6.12 | `cold_cache` (`slow`) | same insights request at 1 rps for 150 s; count samples > 3× the median | — | degraded ≥ **2** spikes (a 60 s TTL over 150 s); baseline ≤ 1 |
|
||
| 6.13 | `poison_data` | request against each `poison_account_ids` entry | — | degraded produces ≥ 1 response with status **500**; baseline produces zero 500s across the same requests (4xx or 200 only) |
|
||
| 6.14 | `chatty_health` | `/healthz` hits on downstream containers during a single transaction, counted from logs | — | degraded ≥ **5** healthz lines across downstreams per transaction; baseline == 0 |
|
||
|
||
**Note on log-line counting (6.8, 6.9, 6.14).** At Phase 0 the plain-text log is
|
||
the only cross-service observability surface that exists — which is precisely the
|
||
deficiency this whole project sets out to fix. Implement these helpers behind
|
||
`harness/containers.py::count_log_lines(service, pattern, since)` so there is
|
||
exactly one place to change. **From Phase 3 these assertions should be rewritten
|
||
against structured logs, and from Phase 4 against span counts** — at which point
|
||
they become both simpler and more robust. Note this in `tests/README.md`; it is a
|
||
deliberate teaching artefact, not a shortcut.
|
||
|
||
**Anti-fake note for Agent B.** 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied
|
||
by inserting a `sleep()`: they require, respectively, real repeated queries, a
|
||
real missing index, real unbounded allocation, real consumer slowness, and a real
|
||
unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12)
|
||
are stated as ratios for the same reason — passing them by faking is more work
|
||
than implementing the mechanism described in the Phase 0 spec.
|
||
|
||
---
|
||
|
||
## 7. The measurement harness
|
||
|
||
`harness/load.py` provides one primitive:
|
||
|
||
```python
|
||
async def drive(
|
||
method: str, url_factory: Callable[[], tuple[str, dict]],
|
||
total: int, concurrency: int,
|
||
warmup: int = 20, timeout: float = 15.0,
|
||
) -> LatencyReport
|
||
```
|
||
|
||
`LatencyReport` carries `count, ok, errors, error_rate, status_counts,
|
||
p50, p95, p99, max, throughput_rps, wall_seconds`, plus a `__str__` suitable for
|
||
dropping straight into an assertion message.
|
||
|
||
Rules:
|
||
|
||
- Warm-up requests are issued and discarded before timing starts.
|
||
- Percentiles from the raw sorted sample, nearest-rank. No interpolation, no
|
||
averaging of percentiles across runs — **median of the p95s**, never the mean.
|
||
- A closed-loop driver (fixed concurrency) is used deliberately for its
|
||
simplicity here. Note in `tests/README.md` that this under-measures latency
|
||
under saturation (coordinated omission) and that Phase 6 replaces it with k6's
|
||
arrival-rate executors. Don't try to fix it in the test suite.
|
||
- `measure_median(n=3, fn)` helper for the pathology tests.
|
||
|
||
`harness/pg.py`:
|
||
|
||
```python
|
||
@contextmanager
|
||
def statement_count(dsn) -> Counter # resets pg_stat_statements, yields, returns delta
|
||
def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON)
|
||
def plan_node_types(plan: dict) -> set[str]
|
||
```
|
||
|
||
`harness/streams.py`: `group_lag(group)`, `wait_for_drain(group, timeout)`.
|
||
|
||
`harness/containers.py`: `rss_mb(service)`, `count_log_lines(service, pattern,
|
||
since)`, `stop(service)`, `start(service)`, `wait_ready(service, timeout)`.
|
||
|
||
---
|
||
|
||
## 8. Fixtures (`conftest.py`)
|
||
|
||
| Fixture | Scope | Purpose |
|
||
|---|---|---|
|
||
| `stack_up` | session | Assert the Compose stack is running and all six services are ready; **fail fast with a clear message** ("run `make up && make seed` first") rather than timing out |
|
||
| `seeded` | session | Assert the seed manifest exists and counts are non-zero; fail fast otherwise |
|
||
| `manifest` | session | Parsed seed manifest |
|
||
| `api` | function | `httpx.AsyncClient` against the gateway, 15 s timeout |
|
||
| `dsn` | session | Postgres DSN from env |
|
||
| `redis_client` | session | — |
|
||
| `chaos` | function | `chaos.set_profile(name)`, `chaos.set_flag(name, bool)`, `chaos.get()`. **Autouse teardown resets to `clean`**, even on failure |
|
||
| `fresh_balances` | function | Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs |
|
||
|
||
Every environment value (ports, DSN, Redis URL) comes from env with the Compose
|
||
defaults as fallbacks. No hardcoded values buried in test bodies.
|
||
|
||
---
|
||
|
||
## 9. `tests/README.md` — what Agent A must write
|
||
|
||
- How to run each marker set, and the prerequisite (`make up && make seed`).
|
||
- The measurement methodology and why ratios are used rather than absolutes.
|
||
- The known-flaky list: 6.10 and 6.12 are the least stable; document the retry
|
||
strategy and what a genuine failure looks like versus noise.
|
||
- The coordinated-omission caveat on the closed-loop driver.
|
||
- The Phase 3/4 migration note for the log-counting helpers.
|
||
- An explicit statement that `test_no_observability.py` is deleted at Phase 2.
|
||
|
||
---
|
||
|
||
## 10. Acceptance criteria
|
||
|
||
For **Agent A**:
|
||
|
||
1. `pytest --collect-only` succeeds and every test carries exactly one goal ID in
|
||
its docstring and at least one marker.
|
||
2. `grep -rn "from services\|from common\|import services\|import common" tests/`
|
||
returns nothing.
|
||
3. Every §5 and §6 item has a corresponding test; `tests/README.md` contains the
|
||
goal-to-test table.
|
||
4. The suite is **red** against an empty repo, and `tests/RED-BASELINE.md`
|
||
records the failure reason per test. No test passes vacuously.
|
||
5. `make test-contract` completes (failing) in under 60 s.
|
||
|
||
For **Agent B**:
|
||
|
||
1. `make test` green three consecutive runs.
|
||
2. `make test-slow` green (may be run once).
|
||
3. `git diff --stat tests/` against Agent A's commit is empty.
|
||
4. `docs/PATHOLOGIES.md` written, and not summarised in the final response.
|