Add project documentation and Claude instructions
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
This commit is contained in:
@@ -0,0 +1,127 @@
|
||||
# Phase 0 — Build the base project, test-first
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`intro.md`](intro.md) · Next: [`phase-1.md`](phase-1.md)*
|
||||
|
||||
**Status:** Done
|
||||
|
||||
**Message:** Both halves complete and verified against the live stack, not just
|
||||
collected.
|
||||
|
||||
- **0a (test suite):** 81 tests across `tests/` (11 files + a 5-module
|
||||
harness), covering all eleven goals G1–G11 and all 13 named pathologies.
|
||||
Verified genuinely red against the empty repo before any implementation
|
||||
existed; `tests/RED-BASELINE.md` records that run. `tests/README.md`
|
||||
documents the invented request-contract fields, measurement methodology,
|
||||
and known-flaky tests.
|
||||
- **0b (implementation):** all seven services (`gateway`, `ledger`, `fraud`,
|
||||
`sanctions`, `insights`, `notifier`, `audit`), `db/schema.sql` +
|
||||
`optional_indexes.sql`, `scripts/seed.py` + `smoke.py`, the frontend, and
|
||||
`docker-compose.yml`. Seeded: 800 accounts, 250,000 transactions, 45
|
||||
merchant categories, 12,000 sanctions entries, in ~12s. `make smoke` and
|
||||
`make test-contract` (64/64) pass; `make test-baseline` and
|
||||
`make test-pathology` pass except for a handful of tests documented below
|
||||
as load-sensitive rather than broken.
|
||||
- **Two changes to the frozen test contract, both made only after flagging
|
||||
them and getting an explicit go-ahead — not applied unilaterally:**
|
||||
- `test_row_counts_match_manifest_and_targets` loosened the `transactions`
|
||||
row-count check from exact equality to `>=`, since other contract tests
|
||||
in the same `make test-contract` run legitimately insert rows first.
|
||||
- `test_pool_starvation_plateaus_throughput` was **removed outright**.
|
||||
Its baseline assertion (throughput scales ≥2.5x from concurrency 5→25)
|
||||
is incompatible with this same spec's own §3 mandate of one uvicorn
|
||||
worker per service — confirmed by reproducing the identical
|
||||
non-scaling behaviour on a bare `GET /healthz` with zero backend
|
||||
logic. `pool_starvation` itself is still fully implemented
|
||||
(`services/ledger/db.py`'s dual-engine pool selection); only that one
|
||||
differential test was dropped.
|
||||
- **Known-flaky, documented rather than papered over:**
|
||||
`test_clean_baseline_is_healthy`, `test_nightmare_degrades_but_survives`,
|
||||
the baseline half of `test_blocking_event_loop_serializes_concurrent_requests`,
|
||||
and `test_missing_index_forces_seq_scan` compare a fast-path measurement
|
||||
against a slow-path one under real concurrent load, and pass at roughly a
|
||||
60–80% rate on ordinary dev hardware depending on momentary host
|
||||
contention. See "Known-flaky tests" in `tests/README.md`.
|
||||
- **Notable implementation deviations** (none change observable behaviour of
|
||||
the 13 pathologies or the request/response contract): gateway's `GET /`
|
||||
serves the frontend rather than §4's generic info JSON (the other five
|
||||
HTTP services still return that JSON as specified); `insights` splits a
|
||||
cheap PK-only `quick_summary` (embedded in every transaction response)
|
||||
from the real cached/N+1-affected 120-day analytics endpoint, so the
|
||||
missing-index pathology doesn't tax every request; `retry_storm` fires
|
||||
all 4 attempts unconditionally rather than stopping at first success
|
||||
(spec's "no circuit breaker" taken literally — also the only way to reach
|
||||
the required lookup-multiplication ratio given `sanctions_errors`' fixed
|
||||
3% rate); `poison_data` resolves via a Redis hash populated at seed time
|
||||
instead of a live table scan, to avoid the deliberately missing index;
|
||||
uvicorn's access log is disabled fleet-wide (redundant with the spec's
|
||||
own logging design, and was double-counting two log-line-based pathology
|
||||
tests); Postgres `max_connections` raised from 100 to 300 (five
|
||||
services' pool sizes can sum past the default under load).
|
||||
|
||||
**Time-box:** 2–3 evenings. Two agent sessions plus one evening of your own
|
||||
reading.
|
||||
|
||||
**Deliverable:** the working Ledgerline system with no observability, plus a
|
||||
black-box integration suite that pins down both its working state and its
|
||||
deliberate brokenness.
|
||||
|
||||
### Why TDD here, when the app itself doesn't matter
|
||||
|
||||
The microservices are throwaway. The *properties* aren't. Everything from Phase 2
|
||||
onward assumes the system still behaves a specific way: seven real services in
|
||||
the path, async work genuinely off the request path, five hot accounts owning a
|
||||
third of the rows, and thirteen pathologies that actually degrade something. You
|
||||
will be editing all seven services for weeks. Without a suite that fails when one
|
||||
of those properties quietly breaks, you'll spend an evening chasing a trace
|
||||
anomaly that turns out to be a regression you introduced yesterday.
|
||||
|
||||
Writing the tests first also forces the contract to be explicit before anything
|
||||
is built — which is what makes a second agent able to implement against it
|
||||
without guessing.
|
||||
|
||||
### 0a — the test suite
|
||||
|
||||
Agent session one, from `phase-0a-test-suite-spec.md`. That agent writes only
|
||||
`tests/` and must not create so much as a stub under `services/`. Its output
|
||||
should be **red**, with a `tests/RED-BASELINE.md` recording why each test fails
|
||||
right now — a test that passes against an empty repo is a broken test.
|
||||
|
||||
The suite asserts goals, not mechanics. Two representative examples worth
|
||||
understanding before you read the code:
|
||||
|
||||
- *"Stop the `sanctions` container and transactions must fail."* This is the test
|
||||
that catches an implementation quietly collapsing seven services into one
|
||||
process. Nothing else would.
|
||||
- *"With `n_plus_one` on, one insights request issues ≥ 50 SQL statements."*
|
||||
Measured via `pg_stat_statements`, not wall clock — because a query count
|
||||
can't be faked with a `sleep()`, and a latency threshold can.
|
||||
|
||||
That second point is the design principle throughout: **prefer assertions that
|
||||
are hard to satisfy without the real mechanism.** Query counts, `EXPLAIN` plan
|
||||
node types, Redis consumer-group lag, container RSS, and genuine unhandled
|
||||
exception paths all have that property.
|
||||
|
||||
### 0b — the implementation
|
||||
|
||||
Agent session two, from `phase-0-base-project-spec.md`, with `tests/` handed over
|
||||
as a **read-only contract**. That agent may not edit, skip, `xfail` or loosen a
|
||||
single test. If something looks impossible it stops and reports rather than
|
||||
adjusting the target.
|
||||
|
||||
Done when `make test` is green three consecutive runs and
|
||||
`git diff --stat tests/` is empty.
|
||||
|
||||
### Then, yourself
|
||||
|
||||
1. **Read every service's `app.py` end to end.** You need to know this codebase
|
||||
cold, because from Phase 2 onward you're modifying all seven services.
|
||||
2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
|
||||
then try to figure out what's wrong using only `docker compose logs`.** Give it
|
||||
twenty real minutes. You will fail, and the specific way you fail — no timing,
|
||||
no causality, no cross-service correlation, a `X-Request-Id` that dies at the
|
||||
Redis Stream — is the motivation for everything that follows. Write down what
|
||||
you wished you had. That list is your acceptance criteria for Phase 8.
|
||||
|
||||
**Done when:** `make test` is green three runs in a row, all nine acceptance
|
||||
criteria in the base spec pass, and you have that written list of frustrations.
|
||||
|
||||
Reference in New Issue
Block a user