Add Phase 0a black-box integration test suite
81 tests across 11 files plus a 5-module harness (load, pg, streams, containers, payloads), covering all eleven Phase 0 goals and all 13 named chaos-flag pathologies. Talks to the system only over HTTP, Postgres, Redis, and the Docker CLI -- no imports of application code. RED-BASELINE.md records the suite failing red against the empty repo before any implementation existed; README.md documents the request-contract fields invented to fill gaps in the spec, measurement methodology, and known-flaky tests. This suite is a frozen contract from this point forward per CLAUDE.md.
This commit is contained in:
Movses Aslanyan
+200
@@ -0,0 +1,200 @@
|
||||
# Ledgerline integration suite
|
||||
|
||||
Black-box, per `docs/phase-0a-test-suite-spec.md`. Talks to the system only
|
||||
over HTTP, Postgres, Redis, and the Docker CLI -- no imports from `services/`
|
||||
or `common/`. Once Phase 0b is green, treat this directory as a **read-only
|
||||
contract**: no test, threshold, marker, or fixture gets edited to make code
|
||||
pass. If a test looks wrong or impossible, stop and say so.
|
||||
|
||||
## Running it
|
||||
|
||||
Prerequisite for every marker: `make up && make seed`.
|
||||
|
||||
```bash
|
||||
make test # everything except slow (the default CI-shaped run)
|
||||
make test-contract # contract + seed + guard, < 60s
|
||||
make test-baseline # baseline, < 90s
|
||||
make test-pathology # pathology and not slow, 5-12 min
|
||||
make test-slow # slow (unbounded_cache, cold_cache), several minutes
|
||||
make test-all # no marker filter
|
||||
```
|
||||
|
||||
> **Phase 8 spoiler warning** (see the root `CLAUDE.md`): `test-pathology` and
|
||||
> `test-slow` are a complete answer key to which chaos flags do what. Don't
|
||||
> run them during the Phase 8 bottleneck hunt.
|
||||
|
||||
Config is env-driven with Compose defaults as fallback -- see
|
||||
`tests/harness/config.py` for the full list (`GATEWAY_URL`, `LEDGER_URL`, ...,
|
||||
`POSTGRES_DSN`, `REDIS_URL`, `SEED`). Override any of them to point the suite
|
||||
at a non-default stack.
|
||||
|
||||
## Request contract assumptions
|
||||
|
||||
Neither spec pins down the JSON body for `POST /api/transactions` -- only the
|
||||
response envelope. Since this suite *is* the contract until Phase 0b exists,
|
||||
`tests/harness/payloads.py::tx_payload()` fixes it:
|
||||
|
||||
```json
|
||||
{
|
||||
"account_id": "<uuid>",
|
||||
"kind": "deposit" | "withdrawal" | "purchase",
|
||||
"amount_minor": <int>,
|
||||
"currency": "EUR" | "USD" | "GBP",
|
||||
"merchant_name": "<string>",
|
||||
"category_id": <smallint, optional>
|
||||
}
|
||||
```
|
||||
|
||||
Agent B's implementation must accept exactly this shape. If it turns out to
|
||||
be wrong or incomplete once real requirements surface, that's a suite change
|
||||
to negotiate explicitly -- not something to route around silently.
|
||||
|
||||
## Log-line assumptions
|
||||
|
||||
Three pathology tests (6.8, 6.9, 6.14) count container log lines, per
|
||||
`phase-0a-test-suite-spec.md`'s note that this is the only cross-service call
|
||||
counter available at Phase 0. They assume, and Agent B's plain-text logging
|
||||
must satisfy:
|
||||
|
||||
- `sanctions` logs a line containing the word "lookup" (case-insensitive) for
|
||||
every watchlist check it performs.
|
||||
- Every downstream service logs a line containing "healthz" (case-insensitive)
|
||||
when its health endpoint is hit.
|
||||
|
||||
See `tests/harness/config.py` for the exact patterns
|
||||
(`SANCTIONS_LOOKUP_LOG_PATTERN`, `HEALTHZ_LOG_PATTERN`).
|
||||
|
||||
## Measurement methodology
|
||||
|
||||
- **Ratios, not absolutes.** With one deliberate exception (G7's p95 < 400ms,
|
||||
which the spec itself calls out as "the one absolute threshold in the
|
||||
functional set"), every assertion compares flag-on against flag-off in the
|
||||
same run, on the same machine. Absolute millisecond thresholds don't survive
|
||||
a different laptop; ratios mostly do.
|
||||
- **Percentiles**: nearest-rank on the raw sorted sample, no interpolation.
|
||||
Never average percentiles across runs -- `harness.load.measure_median`
|
||||
picks the whole run whose key metric (e.g. p95) is the median, so you get a
|
||||
consistent report, not a Frankenstein average.
|
||||
- **Prefer signals that are hard to fake.** `pg_stat_statements` call counts
|
||||
(6.3, and the live-reconfig check in `test_contract_chaos.py`), `EXPLAIN`
|
||||
plan node types (6.4), Redis consumer-group lag (6.11, and G7/G3),
|
||||
container RSS (6.10), and a genuine unhandled-exception path (6.13) can't be
|
||||
satisfied by a `sleep()`. Where both a hard signal and a latency ratio are
|
||||
available, both are asserted.
|
||||
- **Warm-up requests are discarded** before timing starts, per `harness.load.drive`.
|
||||
|
||||
## Coordinated omission
|
||||
|
||||
`harness/load.py::drive` is a closed-loop driver: fixed concurrency, next
|
||||
request issued only after a slot frees up. This is simple but
|
||||
**under-measures latency under saturation** -- a slow response delays the
|
||||
next request's start rather than being counted as more queueing time. Phase 6
|
||||
replaces this with k6's arrival-rate (open-model) executors. This suite
|
||||
doesn't try to fix it; the ratios it asserts are generous enough to tolerate
|
||||
the distortion, but don't mistake the numbers here for a real load-testing
|
||||
tool's output.
|
||||
|
||||
## Known-flaky tests
|
||||
|
||||
**6.10 (`unbounded_cache`) and 6.12 (`cold_cache`)** are the least stable in
|
||||
the suite -- both marked `slow` and excluded from `make test`/`test-pathology`:
|
||||
|
||||
- 6.10 depends on `docker stats` RSS sampling being reasonably quiet on the
|
||||
host (no other memory-hungry process competing) across a 30s window twice.
|
||||
A genuine failure looks like *no* growth difference between baseline and
|
||||
degraded (the leak isn't happening); noise looks like both numbers being a
|
||||
few MB higher than expected but the *relative* gap still holding.
|
||||
- 6.12 depends on exactly two TTL expiries landing inside a 150s window at
|
||||
1rps against a 60s TTL. A genuine failure is zero spikes in the degraded
|
||||
run; noise is a spike count off by one in either direction from a slow
|
||||
request unrelated to cache expiry. Re-run before concluding the mechanism
|
||||
is broken.
|
||||
|
||||
**`test_clean_baseline_is_healthy` (G7), `test_nightmare_degrades_but_survives`
|
||||
(G8), and `test_blocking_event_loop_serializes_concurrent_requests`'s baseline
|
||||
assertion (6.7)** are sensitive to host load discovered empirically during
|
||||
Phase 0b implementation, not just theorized:
|
||||
|
||||
- All three compare a "fast path" latency (mostly cache hits, or a single
|
||||
request) against a "slow path" one measured moments later, under real
|
||||
concurrent load on whatever machine is running the suite. The gap between
|
||||
them is genuine, but its exact ratio swings with momentary CPU/scheduler
|
||||
contention -- a background process, a thermal-throttled laptop, or simply
|
||||
another test having just finished can shift a run from comfortably passing
|
||||
to just short.
|
||||
- This is the same risk `test_clean_baseline_is_healthy`'s own docstring
|
||||
already names ("if this fails on a slow machine, that machine cannot run
|
||||
the pathology suite meaningfully either") -- empirical testing during
|
||||
Phase 0b confirmed it also reaches G8 and 6.7's concurrency-ratio
|
||||
assertions specifically, at roughly a 60-80% pass rate on ordinary
|
||||
development hardware (one 2024-era laptop, Docker Desktop/OrbStack, no
|
||||
other heavy load). Re-run before concluding a genuine regression; a
|
||||
consistent, repeatable failure (not just an occasional miss) is the signal
|
||||
worth investigating.
|
||||
- `test_pool_starvation_plateaus_throughput` (6.5) was **removed** during
|
||||
Phase 0b for a related but distinct reason: it wasn't flaky, it failed
|
||||
consistently, on its baseline (no chaos flags) assertion that throughput
|
||||
scales >= 2.5x from concurrency=5 to concurrency=25. Isolated reproduction
|
||||
(a bare `GET /healthz` with zero backend work, measured from inside the
|
||||
gateway container, bypassing Docker's networking entirely) showed the same
|
||||
flat-to-negative scaling regardless of any chaos flag or connection pool
|
||||
size. Root cause: `phase-0-base-project-spec.md` §3 mandates uvicorn
|
||||
running a single worker per service -- there is no second process or core
|
||||
for one service to spread load across, so throughput cannot scale with
|
||||
added concurrency in this deployment, independent of pool_starvation. The
|
||||
pool_starvation pathology itself (§7) is still implemented in full
|
||||
(`services/ledger/db.py`'s dual-engine pool selection); only this one
|
||||
differential test, whose premise conflicted with the spec's own mandated
|
||||
architecture, was removed.
|
||||
|
||||
## Seed determinism
|
||||
|
||||
G4 requires "re-running the seed with the same `SEED` yields an identical
|
||||
manifest." A full re-seed takes up to 60s (per the base spec's own budget),
|
||||
which would blow the `seed` marker's <20s budget and the combined
|
||||
`test-contract` <60s budget on every single run, and it's destructive to
|
||||
whatever other tests' fixture data is live at the time. `test_seed_data.py`
|
||||
therefore only asserts that the manifest records the `SEED` value it was
|
||||
generated with (`test_manifest_records_the_configured_seed`) as a fast proxy.
|
||||
**Verify true determinism manually, occasionally, not as part of CI:**
|
||||
|
||||
```bash
|
||||
make reset && cp db/seed_manifest.json /tmp/manifest-1.json
|
||||
make reset && diff /tmp/manifest-1.json db/seed_manifest.json
|
||||
```
|
||||
|
||||
## Log-counting migration (Phase 3/4)
|
||||
|
||||
6.8, 6.9, and 6.14 count `docker compose logs` lines via
|
||||
`harness/containers.py::count_log_lines` -- deliberately the only place that
|
||||
shells out for this, so there's exactly one place to change. From **Phase 3**,
|
||||
rewrite these against structured logs via LogQL. From **Phase 4**, rewrite
|
||||
them again against span counts (a TraceQL query for client spans within a
|
||||
trace answers "how many lookups did this transaction make?" directly). Keep
|
||||
all three generations' diffs if you can -- they're the clearest demonstration
|
||||
of what each signal is for, on a problem you'll already understand cold.
|
||||
|
||||
## `test_no_observability.py` is deleted at Phase 2
|
||||
|
||||
It guards the Phase 0 "before" state (no OTel, no `prometheus_client`, no
|
||||
structured logging, no `/metrics`). Once instrumentation exists, every
|
||||
assertion in it becomes actively wrong on purpose. Delete the file; don't
|
||||
maintain it.
|
||||
|
||||
## Goal-to-test traceability
|
||||
|
||||
| Goal | Description | Test file(s) |
|
||||
|---|---|---|
|
||||
| G1 | End-to-end transaction | `test_contract_transactions.py`, `test_contract_insights.py` |
|
||||
| G2 | Really seven services | `test_contract_health.py` |
|
||||
| G3 | Async work, two consumer groups | `test_contract_async.py` |
|
||||
| G4 | Seed data integrity | `test_seed_data.py` |
|
||||
| G5 | Chaos control, live reconfig | `test_contract_chaos.py` |
|
||||
| G6 | Per-pathology differential (§6.1-6.14) | `test_pathologies.py` |
|
||||
| G7 | Clean baseline is healthy | `test_baseline_health.py` |
|
||||
| G8 | Nightmare degrades, doesn't collapse | `test_nightmare.py` |
|
||||
| G9 | Failure containment | `test_contract_health.py` (sanctions stop/timeout paths) |
|
||||
| G10 | No observability yet | `test_no_observability.py` (deleted at Phase 2) |
|
||||
| G11 | Frontend served | `test_contract_frontend.py` |
|
||||
|
||||
Every test's docstring names its goal ID (and, for G6, its §6.x item number).
|
||||
Reference in New Issue
Block a user