# Phase 0 — Technical Spec: "Ledgerline" base project > **How to use this file:** paste the whole thing into your coding agent as the task > spec. It is written to be executed without further clarification. Read > §2 (Non-goals) yourself before you start — the deliberate omissions are the > point of the exercise. --- ## 1. Mission Build **Ledgerline**, a deliberately imperfect microservice system that simulates a small payments/finance backend. It exists for one reason: to be a realistic subject for observability work in later phases. The business logic is fake and does not need to be correct, defensible, or secure. What it *does* need to be is **structurally realistic** — real network hops, real database queries, real async message passing, real contention, real latency distributions, and real failure modes. The system must be runnable with `make up && make seed && make smoke` on a laptop. --- ## 2. Non-goals — read carefully These are **hard constraints**, not preferences. Violating them destroys the value of the project. - **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client, no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no APM agent, no Sentry. Later phases add all of this by hand; that is the lesson. - **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add indexes, connection-pool tuning, batching, backoff, or caching beyond what is specified. - **DO NOT add authentication, authorization, rate limiting, or input hardening.** A single hardcoded demo user is fine. - **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the deliverable. - **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only. - **DO NOT abstract prematurely.** Duplication across services is acceptable and preferred over a clever shared framework. The only logging permitted is Python's standard `logging` module with a plain-text formatter (§9). --- ## 3. Technology | Concern | Choice | |---|---| | Language | Python 3.12 | | Web framework | FastAPI + `uvicorn` (one worker per service) | | HTTP client | `httpx` (async) | | DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` | | Cache / queue | Redis 7 (`redis-py` asyncio client) | | Async messaging | Redis **Streams** (not Celery, not RabbitMQ) | | Validation | Pydantic v2 | | Packaging | One shared `Dockerfile`, one image, `CMD` differs per service | | Orchestration | Docker Compose | | Schema | A single idempotent `db/schema.sql` — no Alembic | | Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework | Redis Streams is chosen deliberately: it has **no automatic context propagation**, so a later phase has to implement it manually. Do not swap it for something with built-in instrumentation. --- ## 4. Services Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`. | # | Service | Port | Role | Stores | |---|---|---|---|---| | 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) | | 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres | | 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) | | 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) | | 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis | | 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres | | 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres | Every HTTP service exposes: - `GET /healthz` → `{"status": "ok", "service": ""}` — liveness, no dependency checks - `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable - `GET /` → `{"service": "", "version": "0.1.0"}` --- ## 5. Domain model (`db/schema.sql`) ``` accounts (id UUID pk, holder_name text, currency char(3), balance_minor bigint, is_hot boolean default false, created_at timestamptz) merchant_categories (id smallint pk, code text unique, label text, risk_weight numeric(3,2)) transactions (id UUID pk, account_id UUID, kind text, -- deposit|withdrawal|purchase amount_minor bigint, currency char(3), merchant_name text, category_id smallint, status text, -- pending|settled|rejected risk_score numeric(4,3), created_at timestamptz) sanctions_entries (id serial pk, normalized_name text, country char(2), list_source text, added_at timestamptz) notifications (id UUID pk, transaction_id UUID, channel text, -- email|push|sms status text, attempts int, sent_at timestamptz) audit_events (id bigserial pk, transaction_id UUID, event_type text, payload jsonb, recorded_at timestamptz) ``` **Index policy — important.** Create only these indexes: - primary keys (implicit) - `accounts(holder_name)` - `sanctions_entries(normalized_name)` Deliberately **omit** `transactions(account_id, created_at)` and `transactions(category_id)`. Put the `CREATE INDEX` statements for those in `db/optional_indexes.sql`, applied only by `make fix-index`. --- ## 6. Request flows ### 6.1 `POST /api/transactions` — the primary flow The trace this produces should be ~4 levels deep with one async fork. ``` browser └─ gateway POST /api/transactions ├─ ledger POST /internal/transactions (create as `pending`, row lock on account) ├─ fraud POST /internal/score (risk scoring) │ └─ sanctions POST /internal/lookup (watchlist check — slow vendor) ├─ ledger POST /internal/transactions/{id}/settle (or /reject if risk_score > 0.8) ├─ redis XADD stream:transactions (fire-and-forget) │ ├─ notifier consumer group `notifier` │ └─ audit consumer group `audit` └─ insights GET /internal/accounts/{id}/summary (cached) ``` Response body: ```json { "transaction_id": "…", "status": "settled", "risk_score": 0.12, "balance_minor": 481200, "summary": { "…": "…" }, "request_id": "…", "elapsed_ms": 843 } ``` ### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow `gateway → insights`. `insights` runs a 120-day aggregation over `transactions`. This is where the N+1 and the missing index bite. ### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown ### 6.4 Admin - `GET /api/admin/chaos` → current flag values - `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}` - `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}` Flags live in a Redis hash `chaos:flags` so every service reads the same state without a restart. Each service reads the hash on **every request** (a tiny bit of extra Redis load is fine and realistic). --- ## 7. The pathology catalogue Each pathology lives in a clearly named `chaos.py` module inside its owning service, with a docstring naming it. All flags default **off**. | Flag | Service | Behaviour when enabled | |---|---|---| | `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). | | `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. | | `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. | | `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) | | `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. | | `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. | | `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. | | `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. | | `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. | | `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. | | `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. | | `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. | | `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. | **Profiles** - `clean` — all off. - `light` — `slow_sanctions`, `cold_cache`. - `nightmare` — all thirteen on. - `random` — pick a random subset of size 3–6, using a seed derived from `time.time()` and **not logged anywhere**. Store the active set in Redis under `chaos:flags` only, never write it to a file or print it. (This makes the later bottleneck-hunting exercise a genuine hunt.) Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the symptom a reader should expect. Do **not** reproduce or summarise its contents in your final chat response. --- ## 8. Seed data (`scripts/seed.py`) Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert. - **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD, 10% GBP. Balances log-normal, median ≈ 250 000 minor units. Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets. - **45 merchant categories** with `risk_weight` between 0.01 and 0.95. - **250 000 transactions** spread over the last 120 days. - Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip. - Amounts log-normal. - Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest on a Zipf-ish distribution. - `status` 92% `settled`, 5% `rejected`, 3% `pending`. - **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account holders, so a meaningful fraction of requests genuinely score as high risk. - **~2% malformed rows** distributed through `transactions`: - `currency` = `NULL` - negative `amount_minor` on a `deposit` - `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence - `created_at` set 3 days in the future - `category_id` pointing at a non-existent category Print a summary table at the end (row counts per table, hot account IDs, count of malformed rows). Seeding 250k rows must complete in under 60 seconds — use `COPY` / `executemany`, not per-row inserts. --- ## 9. Logging (the deliberately bad baseline) Every service configures: ```python logging.basicConfig( level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s", ) ``` - Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value. - `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it as a header on every downstream `httpx` call. Each service logs it manually at the start of each handler. **This is the only correlation mechanism that exists.** It is intentionally weak — it does not survive the Redis Stream hop, it carries no timing, and it cannot express parent/child relationships. - Set `PYTHONUNBUFFERED=1` in the Dockerfile. - Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK. --- ## 10. Frontend Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`, `app.js`, `style.css`. No bundler, no npm, no CDN dependencies. Layout, top to bottom: 1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`), amount input, merchant name input, kind radio (deposit / withdrawal / purchase), Submit button. 2. **Result panel** — pretty-printed JSON response, plus the browser-measured round-trip time in ms, plus the `request_id` shown in a monospace font that is easy to copy. 3. **Load burst** — a button that fires *N* concurrent requests (N selectable: 10 / 50 / 200) against random accounts, then renders a small table: count, success, errors, min / mean / p50 / p95 / max ms, measured client-side. This is a crude stand-in for a real load tool; a later phase replaces it with k6. 4. **Chaos panel** — a checkbox per flag reflecting current state from `GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile buttons (`clean`, `light`, `nightmare`, `random`). 5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green or red dot per service. Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS. --- ## 11. Repository layout ``` ledgerline/ ├─ Makefile ├─ docker-compose.yml ├─ Dockerfile # single image for all Python services ├─ pyproject.toml # or requirements.txt — one dependency set ├─ .env.example ├─ README.md ├─ db/ │ ├─ schema.sql │ └─ optional_indexes.sql ├─ scripts/ │ ├─ seed.py │ └─ smoke.py ├─ common/ │ ├─ __init__.py │ ├─ settings.py # pydantic-settings, env-driven │ ├─ db.py # async engine/session factory │ ├─ redis_client.py │ ├─ http.py # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough │ └─ logging_config.py # the plain-text logging setup from §9 ├─ services/ │ ├─ gateway/ (app.py, chaos.py, orchestrator.py, static/) │ ├─ ledger/ (app.py, chaos.py, repository.py) │ ├─ fraud/ (app.py, chaos.py, scoring.py) │ ├─ sanctions/ (app.py, chaos.py, repository.py) │ ├─ insights/ (app.py, chaos.py, analytics.py) │ ├─ notifier/ (app.py, chaos.py, consumer.py) │ └─ audit/ (worker.py, consumer.py) └─ docs/ └─ PATHOLOGIES.md ``` `docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use `depends_on` with `condition: service_healthy` for postgres and redis. Mount the source read-write and run uvicorn with `--reload` so iteration is fast. Reserve — but leave empty and unreferenced — a top-level `observability/` directory with a `.gitkeep`. Later phases fill it. --- ## 12. Makefile targets | Target | Effect | |---|---| | `make up` / `make down` | compose up -d / down | | `make logs` | `docker compose logs -f` | | `make seed` | apply `db/schema.sql`, run `scripts/seed.py` | | `make reset` | `down -v`, `up`, `seed` | | `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s | | `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile | | `make fix-index` | apply `db/optional_indexes.sql` | | `make drop-index` | drop them again | --- ## 13. Acceptance criteria The task is done when all of these hold on a clean checkout: 1. `make reset` completes in under 3 minutes and prints the seed summary table. 2. `make smoke` passes with `chaos-clean`. 3. The frontend loads at `http://localhost:8000`, the account dropdown is populated, and a submitted transaction returns a settled or rejected result. 4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under ~250 ms and zero errors. 5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error rate between 2% and 10% — degraded but not fully broken. 6. Toggling any single flag via the chaos panel changes behaviour **without a restart**. 7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`. 8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns nothing. 9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies. In your final message, report only: services started, row counts seeded, the p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.