Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
374 lines
17 KiB
Markdown
374 lines
17 KiB
Markdown
# Phase 0 — Technical Spec: "Ledgerline" base project
|
||
|
||
> **How to use this file:** paste the whole thing into your coding agent as the task
|
||
> spec. It is written to be executed without further clarification. Read
|
||
> §2 (Non-goals) yourself before you start — the deliberate omissions are the
|
||
> point of the exercise.
|
||
|
||
---
|
||
|
||
## 1. Mission
|
||
|
||
Build **Ledgerline**, a deliberately imperfect microservice system that simulates a
|
||
small payments/finance backend. It exists for one reason: to be a realistic subject
|
||
for observability work in later phases. The business logic is fake and does not
|
||
need to be correct, defensible, or secure. What it *does* need to be is
|
||
**structurally realistic** — real network hops, real database queries, real async
|
||
message passing, real contention, real latency distributions, and real failure modes.
|
||
|
||
The system must be runnable with `make up && make seed && make smoke` on a laptop.
|
||
|
||
---
|
||
|
||
## 2. Non-goals — read carefully
|
||
|
||
These are **hard constraints**, not preferences. Violating them destroys the value
|
||
of the project.
|
||
|
||
- **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client,
|
||
no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no
|
||
APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
|
||
- **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add
|
||
indexes, connection-pool tuning, batching, backoff, or caching beyond what is
|
||
specified.
|
||
- **DO NOT add authentication, authorization, rate limiting, or input hardening.**
|
||
A single hardcoded demo user is fine.
|
||
- **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the
|
||
deliverable.
|
||
- **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only.
|
||
- **DO NOT abstract prematurely.** Duplication across services is acceptable and
|
||
preferred over a clever shared framework.
|
||
|
||
The only logging permitted is Python's standard `logging` module with a plain-text
|
||
formatter (§9).
|
||
|
||
---
|
||
|
||
## 3. Technology
|
||
|
||
| Concern | Choice |
|
||
|---|---|
|
||
| Language | Python 3.12 |
|
||
| Web framework | FastAPI + `uvicorn` (one worker per service) |
|
||
| HTTP client | `httpx` (async) |
|
||
| DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` |
|
||
| Cache / queue | Redis 7 (`redis-py` asyncio client) |
|
||
| Async messaging | Redis **Streams** (not Celery, not RabbitMQ) |
|
||
| Validation | Pydantic v2 |
|
||
| Packaging | One shared `Dockerfile`, one image, `CMD` differs per service |
|
||
| Orchestration | Docker Compose |
|
||
| Schema | A single idempotent `db/schema.sql` — no Alembic |
|
||
| Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework |
|
||
|
||
Redis Streams is chosen deliberately: it has **no automatic context propagation**,
|
||
so a later phase has to implement it manually. Do not swap it for something with
|
||
built-in instrumentation.
|
||
|
||
---
|
||
|
||
## 4. Services
|
||
|
||
Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`.
|
||
|
||
| # | Service | Port | Role | Stores |
|
||
|---|---|---|---|---|
|
||
| 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) |
|
||
| 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
|
||
| 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
|
||
| 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) |
|
||
| 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis |
|
||
| 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres |
|
||
| 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |
|
||
|
||
Every HTTP service exposes:
|
||
|
||
- `GET /healthz` → `{"status": "ok", "service": "<name>"}` — liveness, no dependency checks
|
||
- `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable
|
||
- `GET /` → `{"service": "<name>", "version": "0.1.0"}`
|
||
|
||
---
|
||
|
||
## 5. Domain model (`db/schema.sql`)
|
||
|
||
```
|
||
accounts (id UUID pk, holder_name text, currency char(3),
|
||
balance_minor bigint, is_hot boolean default false,
|
||
created_at timestamptz)
|
||
|
||
merchant_categories (id smallint pk, code text unique, label text,
|
||
risk_weight numeric(3,2))
|
||
|
||
transactions (id UUID pk, account_id UUID, kind text, -- deposit|withdrawal|purchase
|
||
amount_minor bigint, currency char(3),
|
||
merchant_name text, category_id smallint,
|
||
status text, -- pending|settled|rejected
|
||
risk_score numeric(4,3), created_at timestamptz)
|
||
|
||
sanctions_entries (id serial pk, normalized_name text, country char(2),
|
||
list_source text, added_at timestamptz)
|
||
|
||
notifications (id UUID pk, transaction_id UUID, channel text, -- email|push|sms
|
||
status text, attempts int, sent_at timestamptz)
|
||
|
||
audit_events (id bigserial pk, transaction_id UUID, event_type text,
|
||
payload jsonb, recorded_at timestamptz)
|
||
```
|
||
|
||
**Index policy — important.** Create only these indexes:
|
||
|
||
- primary keys (implicit)
|
||
- `accounts(holder_name)`
|
||
- `sanctions_entries(normalized_name)`
|
||
|
||
Deliberately **omit** `transactions(account_id, created_at)` and
|
||
`transactions(category_id)`. Put the `CREATE INDEX` statements for those in
|
||
`db/optional_indexes.sql`, applied only by `make fix-index`.
|
||
|
||
---
|
||
|
||
## 6. Request flows
|
||
|
||
### 6.1 `POST /api/transactions` — the primary flow
|
||
|
||
The trace this produces should be ~4 levels deep with one async fork.
|
||
|
||
```
|
||
browser
|
||
└─ gateway POST /api/transactions
|
||
├─ ledger POST /internal/transactions (create as `pending`, row lock on account)
|
||
├─ fraud POST /internal/score (risk scoring)
|
||
│ └─ sanctions POST /internal/lookup (watchlist check — slow vendor)
|
||
├─ ledger POST /internal/transactions/{id}/settle (or /reject if risk_score > 0.8)
|
||
├─ redis XADD stream:transactions (fire-and-forget)
|
||
│ ├─ notifier consumer group `notifier`
|
||
│ └─ audit consumer group `audit`
|
||
└─ insights GET /internal/accounts/{id}/summary (cached)
|
||
```
|
||
|
||
Response body:
|
||
|
||
```json
|
||
{
|
||
"transaction_id": "…", "status": "settled", "risk_score": 0.12,
|
||
"balance_minor": 481200, "summary": { "…": "…" },
|
||
"request_id": "…", "elapsed_ms": 843
|
||
}
|
||
```
|
||
|
||
### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow
|
||
|
||
`gateway → insights`. `insights` runs a 120-day aggregation over `transactions`.
|
||
This is where the N+1 and the missing index bite.
|
||
|
||
### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown
|
||
|
||
### 6.4 Admin
|
||
|
||
- `GET /api/admin/chaos` → current flag values
|
||
- `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}`
|
||
- `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}`
|
||
|
||
Flags live in a Redis hash `chaos:flags` so every service reads the same state
|
||
without a restart. Each service reads the hash on **every request** (a tiny bit of
|
||
extra Redis load is fine and realistic).
|
||
|
||
---
|
||
|
||
## 7. The pathology catalogue
|
||
|
||
Each pathology lives in a clearly named `chaos.py` module inside its owning service,
|
||
with a docstring naming it. All flags default **off**.
|
||
|
||
| Flag | Service | Behaviour when enabled |
|
||
|---|---|---|
|
||
| `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). |
|
||
| `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. |
|
||
| `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. |
|
||
| `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) |
|
||
| `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. |
|
||
| `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. |
|
||
| `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. |
|
||
| `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. |
|
||
| `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. |
|
||
| `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. |
|
||
| `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. |
|
||
| `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. |
|
||
| `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. |
|
||
|
||
**Profiles**
|
||
|
||
- `clean` — all off.
|
||
- `light` — `slow_sanctions`, `cold_cache`.
|
||
- `nightmare` — all thirteen on.
|
||
- `random` — pick a random subset of size 3–6, using a seed derived from
|
||
`time.time()` and **not logged anywhere**. Store the active set in Redis under
|
||
`chaos:flags` only, never write it to a file or print it. (This makes the later
|
||
bottleneck-hunting exercise a genuine hunt.)
|
||
|
||
Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the
|
||
symptom a reader should expect. Do **not** reproduce or summarise its contents in
|
||
your final chat response.
|
||
|
||
---
|
||
|
||
## 8. Seed data (`scripts/seed.py`)
|
||
|
||
Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert.
|
||
|
||
- **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD,
|
||
10% GBP. Balances log-normal, median ≈ 250 000 minor units.
|
||
Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets.
|
||
- **45 merchant categories** with `risk_weight` between 0.01 and 0.95.
|
||
- **250 000 transactions** spread over the last 120 days.
|
||
- Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
|
||
- Amounts log-normal.
|
||
- Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest
|
||
on a Zipf-ish distribution.
|
||
- `status` 92% `settled`, 5% `rejected`, 3% `pending`.
|
||
- **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account
|
||
holders, so a meaningful fraction of requests genuinely score as high risk.
|
||
- **~2% malformed rows** distributed through `transactions`:
|
||
- `currency` = `NULL`
|
||
- negative `amount_minor` on a `deposit`
|
||
- `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence
|
||
- `created_at` set 3 days in the future
|
||
- `category_id` pointing at a non-existent category
|
||
|
||
Print a summary table at the end (row counts per table, hot account IDs, count of
|
||
malformed rows). Seeding 250k rows must complete in under 60 seconds — use
|
||
`COPY` / `executemany`, not per-row inserts.
|
||
|
||
---
|
||
|
||
## 9. Logging (the deliberately bad baseline)
|
||
|
||
Every service configures:
|
||
|
||
```python
|
||
logging.basicConfig(
|
||
level=logging.INFO,
|
||
format="%(asctime)s %(levelname)s %(name)s %(message)s",
|
||
)
|
||
```
|
||
|
||
- Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value.
|
||
- `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it
|
||
as a header on every downstream `httpx` call. Each service logs it manually at the
|
||
start of each handler. **This is the only correlation mechanism that exists.**
|
||
It is intentionally weak — it does not survive the Redis Stream hop, it carries no
|
||
timing, and it cannot express parent/child relationships.
|
||
- Set `PYTHONUNBUFFERED=1` in the Dockerfile.
|
||
- Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK.
|
||
|
||
---
|
||
|
||
## 10. Frontend
|
||
|
||
Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`,
|
||
`app.js`, `style.css`. No bundler, no npm, no CDN dependencies.
|
||
|
||
Layout, top to bottom:
|
||
|
||
1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`),
|
||
amount input, merchant name input, kind radio (deposit / withdrawal / purchase),
|
||
Submit button.
|
||
2. **Result panel** — pretty-printed JSON response, plus the browser-measured
|
||
round-trip time in ms, plus the `request_id` shown in a monospace font that is
|
||
easy to copy.
|
||
3. **Load burst** — a button that fires *N* concurrent requests (N selectable:
|
||
10 / 50 / 200) against random accounts, then renders a small table:
|
||
count, success, errors, min / mean / p50 / p95 / max ms, measured client-side.
|
||
This is a crude stand-in for a real load tool; a later phase replaces it with k6.
|
||
4. **Chaos panel** — a checkbox per flag reflecting current state from
|
||
`GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile
|
||
buttons (`clean`, `light`, `nightmare`, `random`).
|
||
5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s
|
||
via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green
|
||
or red dot per service.
|
||
|
||
Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.
|
||
|
||
---
|
||
|
||
## 11. Repository layout
|
||
|
||
```
|
||
ledgerline/
|
||
├─ Makefile
|
||
├─ docker-compose.yml
|
||
├─ Dockerfile # single image for all Python services
|
||
├─ pyproject.toml # or requirements.txt — one dependency set
|
||
├─ .env.example
|
||
├─ README.md
|
||
├─ db/
|
||
│ ├─ schema.sql
|
||
│ └─ optional_indexes.sql
|
||
├─ scripts/
|
||
│ ├─ seed.py
|
||
│ └─ smoke.py
|
||
├─ common/
|
||
│ ├─ __init__.py
|
||
│ ├─ settings.py # pydantic-settings, env-driven
|
||
│ ├─ db.py # async engine/session factory
|
||
│ ├─ redis_client.py
|
||
│ ├─ http.py # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
|
||
│ └─ logging_config.py # the plain-text logging setup from §9
|
||
├─ services/
|
||
│ ├─ gateway/ (app.py, chaos.py, orchestrator.py, static/)
|
||
│ ├─ ledger/ (app.py, chaos.py, repository.py)
|
||
│ ├─ fraud/ (app.py, chaos.py, scoring.py)
|
||
│ ├─ sanctions/ (app.py, chaos.py, repository.py)
|
||
│ ├─ insights/ (app.py, chaos.py, analytics.py)
|
||
│ ├─ notifier/ (app.py, chaos.py, consumer.py)
|
||
│ └─ audit/ (worker.py, consumer.py)
|
||
└─ docs/
|
||
└─ PATHOLOGIES.md
|
||
```
|
||
|
||
`docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use
|
||
`depends_on` with `condition: service_healthy` for postgres and redis. Mount the
|
||
source read-write and run uvicorn with `--reload` so iteration is fast.
|
||
|
||
Reserve — but leave empty and unreferenced — a top-level `observability/` directory
|
||
with a `.gitkeep`. Later phases fill it.
|
||
|
||
---
|
||
|
||
## 12. Makefile targets
|
||
|
||
| Target | Effect |
|
||
|---|---|
|
||
| `make up` / `make down` | compose up -d / down |
|
||
| `make logs` | `docker compose logs -f` |
|
||
| `make seed` | apply `db/schema.sql`, run `scripts/seed.py` |
|
||
| `make reset` | `down -v`, `up`, `seed` |
|
||
| `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s |
|
||
| `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile |
|
||
| `make fix-index` | apply `db/optional_indexes.sql` |
|
||
| `make drop-index` | drop them again |
|
||
|
||
---
|
||
|
||
## 13. Acceptance criteria
|
||
|
||
The task is done when all of these hold on a clean checkout:
|
||
|
||
1. `make reset` completes in under 3 minutes and prints the seed summary table.
|
||
2. `make smoke` passes with `chaos-clean`.
|
||
3. The frontend loads at `http://localhost:8000`, the account dropdown is populated,
|
||
and a submitted transaction returns a settled or rejected result.
|
||
4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under
|
||
~250 ms and zero errors.
|
||
5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error
|
||
rate between 2% and 10% — degraded but not fully broken.
|
||
6. Toggling any single flag via the chaos panel changes behaviour **without a
|
||
restart**.
|
||
7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`.
|
||
8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns
|
||
nothing.
|
||
9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies.
|
||
|
||
In your final message, report only: services started, row counts seeded, the
|
||
p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you
|
||
deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.
|