Add Phase 0a black-box integration test suite
81 tests across 11 files plus a 5-module harness (load, pg, streams, containers, payloads), covering all eleven Phase 0 goals and all 13 named chaos-flag pathologies. Talks to the system only over HTTP, Postgres, Redis, and the Docker CLI -- no imports of application code. RED-BASELINE.md records the suite failing red against the empty repo before any implementation existed; README.md documents the request-contract fields invented to fill gaps in the spec, measurement methodology, and known-flaky tests. This suite is a frozen contract from this point forward per CLAUDE.md.
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# RED-BASELINE
|
||||
|
||||
Recorded by running the suite, as written, against the actual state of the
|
||||
repo at the end of Phase 0a: no `services/`, `common/`, `db/`, `scripts/`, or
|
||||
`docker-compose.yml` exist yet, and no containers are running. This is the
|
||||
required proof that nothing in the suite passes vacuously.
|
||||
|
||||
Commands run (Python 3.12 venv, `pip install -r requirements-test.txt`):
|
||||
|
||||
```
|
||||
pytest --collect-only -q # 82 tests collected, 0 errors
|
||||
pytest -m "contract or seed or guard" -q # 64 errors, 18 deselected
|
||||
pytest -m "pathology or slow or baseline" -q # 18 errors, 64 deselected
|
||||
```
|
||||
|
||||
## Why every test currently fails
|
||||
|
||||
**All 82 tests fail at fixture setup with the same root cause: connection
|
||||
refused.** The session-scoped, autouse `stack_up` fixture in `conftest.py`
|
||||
polls `GET /healthz` on all six HTTP services for 30s, gets nothing back
|
||||
(there is no gateway, no ledger, no fraud, no sanctions, no insights, no
|
||||
notifier -- none of `services/` exists), and calls `pytest.fail()` with:
|
||||
|
||||
```
|
||||
Stack is not up: fraud, gateway, insights, ledger, notifier, sanctions did not
|
||||
answer /healthz within 30s. Run `make up && make seed` first.
|
||||
```
|
||||
|
||||
Every other test in the suite depends on `stack_up` (directly, or
|
||||
transitively through `api`, `chaos`, `manifest` -> `seeded`, or `dsn` used
|
||||
against a nonexistent Postgres). pytest caches the session-scoped failure, so
|
||||
after the first 30s timeout every remaining test errors immediately at setup
|
||||
with the same message, rather than re-polling.
|
||||
|
||||
This is deliberately the *first* thing every test checks. It is a stronger
|
||||
and more honest RED signal than a wall of individual connection-refused
|
||||
tracebacks from `httpx`/`psycopg`/`redis`, and it's exactly the failure mode a
|
||||
human would hit running this suite manually against nothing.
|
||||
|
||||
## The one test that would fail on its own merits even with a live stack
|
||||
|
||||
`tests/test_no_observability.py::test_application_code_exists` is a sanity
|
||||
guard, not a real assertion about the system: it exists specifically so
|
||||
`test_no_file_imports_observability_libraries` (which walks `services/` and
|
||||
`common/` looking for forbidden imports) can't pass vacuously by finding zero
|
||||
`.py` files. Right now it fails at the `stack_up` setup step like everything
|
||||
else, but if `stack_up` is bypassed (e.g. by testing this file's guard logic
|
||||
in isolation against the current empty repo), it fails on its own assertion:
|
||||
|
||||
```
|
||||
no .py files found under services/ or common/ -- either the app hasn't been
|
||||
implemented yet, or the guard tests below would pass vacuously
|
||||
```
|
||||
|
||||
That confirms the vacuous-pass guard itself is red for the right reason too.
|
||||
|
||||
## What "green" requires
|
||||
|
||||
`make up && make seed` need `docker-compose.yml`, `db/schema.sql`,
|
||||
`scripts/seed.py`, and all seven `services/*` -- i.e. all of Phase 0b, built
|
||||
from `docs/phase-0-base-project-spec.md` against this suite as a read-only
|
||||
contract.
|
||||
Reference in New Issue
Block a user