LedgerLine/docs/phase-0-base-project-spec.md

# Phase 0 — Technical Spec: "Ledgerline" base project

> **How to use this file:** paste the whole thing into your coding agent as the task
> spec. It is written to be executed without further clarification. Read
> §2 (Non-goals) yourself before you start — the deliberate omissions are the
> point of the exercise.

---

## 1. Mission

Build **Ledgerline**, a deliberately imperfect microservice system that simulates a
small payments/finance backend. It exists for one reason: to be a realistic subject
for observability work in later phases. The business logic is fake and does not
need to be correct, defensible, or secure. What it *does* need to be is
**structurally realistic** — real network hops, real database queries, real async
message passing, real contention, real latency distributions, and real failure modes.

The system must be runnable with `make up && make seed && make smoke` on a laptop.

---

## 2. Non-goals — read carefully

These are **hard constraints**, not preferences. Violating them destroys the value
of the project.

- **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client,
  no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no
  APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
- **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add
  indexes, connection-pool tuning, batching, backoff, or caching beyond what is
  specified.
- **DO NOT add authentication, authorization, rate limiting, or input hardening.**
  A single hardcoded demo user is fine.
- **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the
  deliverable.
- **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only.
- **DO NOT abstract prematurely.** Duplication across services is acceptable and
  preferred over a clever shared framework.

The only logging permitted is Python's standard `logging` module with a plain-text
formatter (§9).

---

## 3. Technology

| Concern | Choice |
|---|---|
| Language | Python 3.12 |
| Web framework | FastAPI + `uvicorn` (one worker per service) |
| HTTP client | `httpx` (async) |
| DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` |
| Cache / queue | Redis 7 (`redis-py` asyncio client) |
| Async messaging | Redis **Streams** (not Celery, not RabbitMQ) |
| Validation | Pydantic v2 |
| Packaging | One shared `Dockerfile`, one image, `CMD` differs per service |
| Orchestration | Docker Compose |
| Schema | A single idempotent `db/schema.sql` — no Alembic |
| Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework |

Redis Streams is chosen deliberately: it has **no automatic context propagation**,
so a later phase has to implement it manually. Do not swap it for something with
built-in instrumentation.

---

## 4. Services

Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`.

| # | Service | Port | Role | Stores |
|---|---|---|---|---|
| 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) |
| 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
| 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
| 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) |
| 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis |
| 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres |
| 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |

Every HTTP service exposes:

- `GET /healthz` → `{"status": "ok", "service": "<name>"}` — liveness, no dependency checks
- `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable
- `GET /` → `{"service": "<name>", "version": "0.1.0"}`

---

## 5. Domain model (`db/schema.sql`)

```
accounts            (id UUID pk, holder_name text, currency char(3),
                     balance_minor bigint, is_hot boolean default false,
                     created_at timestamptz)

merchant_categories (id smallint pk, code text unique, label text,
                     risk_weight numeric(3,2))

transactions        (id UUID pk, account_id UUID, kind text,          -- deposit|withdrawal|purchase
                     amount_minor bigint, currency char(3),
                     merchant_name text, category_id smallint,
                     status text,                                     -- pending|settled|rejected
                     risk_score numeric(4,3), created_at timestamptz)

sanctions_entries   (id serial pk, normalized_name text, country char(2),
                     list_source text, added_at timestamptz)

notifications       (id UUID pk, transaction_id UUID, channel text,   -- email|push|sms
                     status text, attempts int, sent_at timestamptz)

audit_events        (id bigserial pk, transaction_id UUID, event_type text,
                     payload jsonb, recorded_at timestamptz)
```

**Index policy — important.** Create only these indexes:

- primary keys (implicit)
- `accounts(holder_name)`
- `sanctions_entries(normalized_name)`

Deliberately **omit** `transactions(account_id, created_at)` and
`transactions(category_id)`. Put the `CREATE INDEX` statements for those in
`db/optional_indexes.sql`, applied only by `make fix-index`.

---

## 6. Request flows

### 6.1 `POST /api/transactions` — the primary flow

The trace this produces should be ~4 levels deep with one async fork.

```
browser
  └─ gateway  POST /api/transactions
       ├─ ledger   POST /internal/transactions          (create as `pending`, row lock on account)
       ├─ fraud    POST /internal/score                 (risk scoring)
       │     └─ sanctions  POST /internal/lookup        (watchlist check — slow vendor)
       ├─ ledger   POST /internal/transactions/{id}/settle   (or /reject if risk_score > 0.8)
       ├─ redis    XADD  stream:transactions            (fire-and-forget)
       │              ├─ notifier  consumer group `notifier`
       │              └─ audit     consumer group `audit`
       └─ insights GET  /internal/accounts/{id}/summary (cached)
```

Response body:

```json
{
  "transaction_id": "…", "status": "settled", "risk_score": 0.12,
  "balance_minor": 481200, "summary": { "…": "…" },
  "request_id": "…", "elapsed_ms": 843
}
```

### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow

`gateway → insights`. `insights` runs a 120-day aggregation over `transactions`.
This is where the N+1 and the missing index bite.

### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown

### 6.4 Admin

- `GET  /api/admin/chaos` → current flag values
- `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}`
- `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}`

Flags live in a Redis hash `chaos:flags` so every service reads the same state
without a restart. Each service reads the hash on **every request** (a tiny bit of
extra Redis load is fine and realistic).

---

## 7. The pathology catalogue

Each pathology lives in a clearly named `chaos.py` module inside its owning service,
with a docstring naming it. All flags default **off**.

| Flag | Service | Behaviour when enabled |
|---|---|---|
| `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). |
| `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. |
| `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. |
| `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) |
| `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. |
| `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. |
| `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. |
| `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. |
| `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. |
| `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. |
| `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. |
| `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. |
| `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. |

**Profiles**

- `clean` — all off.
- `light` — `slow_sanctions`, `cold_cache`.
- `nightmare` — all thirteen on.
- `random` — pick a random subset of size 3–6, using a seed derived from
  `time.time()` and **not logged anywhere**. Store the active set in Redis under
  `chaos:flags` only, never write it to a file or print it. (This makes the later
  bottleneck-hunting exercise a genuine hunt.)

Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the
symptom a reader should expect. Do **not** reproduce or summarise its contents in
your final chat response.

---

## 8. Seed data (`scripts/seed.py`)

Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert.

- **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD,
  10% GBP. Balances log-normal, median ≈ 250 000 minor units.
  Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets.
- **45 merchant categories** with `risk_weight` between 0.01 and 0.95.
- **250 000 transactions** spread over the last 120 days.
  - Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
  - Amounts log-normal.
  - Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest
    on a Zipf-ish distribution.
  - `status` 92% `settled`, 5% `rejected`, 3% `pending`.
- **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account
  holders, so a meaningful fraction of requests genuinely score as high risk.
- **~2% malformed rows** distributed through `transactions`:
  - `currency` = `NULL`
  - negative `amount_minor` on a `deposit`
  - `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence
  - `created_at` set 3 days in the future
  - `category_id` pointing at a non-existent category

Print a summary table at the end (row counts per table, hot account IDs, count of
malformed rows). Seeding 250k rows must complete in under 60 seconds — use
`COPY` / `executemany`, not per-row inserts.

---

## 9. Logging (the deliberately bad baseline)

Every service configures:

```python
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
```

- Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value.
- `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it
  as a header on every downstream `httpx` call. Each service logs it manually at the
  start of each handler. **This is the only correlation mechanism that exists.**
  It is intentionally weak — it does not survive the Redis Stream hop, it carries no
  timing, and it cannot express parent/child relationships.
- Set `PYTHONUNBUFFERED=1` in the Dockerfile.
- Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK.

---

## 10. Frontend

Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`,
`app.js`, `style.css`. No bundler, no npm, no CDN dependencies.

Layout, top to bottom:

1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`),
   amount input, merchant name input, kind radio (deposit / withdrawal / purchase),
   Submit button.
2. **Result panel** — pretty-printed JSON response, plus the browser-measured
   round-trip time in ms, plus the `request_id` shown in a monospace font that is
   easy to copy.
3. **Load burst** — a button that fires *N* concurrent requests (N selectable:
   10 / 50 / 200) against random accounts, then renders a small table:
   count, success, errors, min / mean / p50 / p95 / max ms, measured client-side.
   This is a crude stand-in for a real load tool; a later phase replaces it with k6.
4. **Chaos panel** — a checkbox per flag reflecting current state from
   `GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile
   buttons (`clean`, `light`, `nightmare`, `random`).
5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s
   via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green
   or red dot per service.

Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.

---

## 11. Repository layout

```
ledgerline/
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile                  # single image for all Python services
├─ pyproject.toml              # or requirements.txt — one dependency set
├─ .env.example
├─ README.md
├─ db/
│  ├─ schema.sql
│  └─ optional_indexes.sql
├─ scripts/
│  ├─ seed.py
│  └─ smoke.py
├─ common/
│  ├─ __init__.py
│  ├─ settings.py              # pydantic-settings, env-driven
│  ├─ db.py                    # async engine/session factory
│  ├─ redis_client.py
│  ├─ http.py                  # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
│  └─ logging_config.py        # the plain-text logging setup from §9
├─ services/
│  ├─ gateway/    (app.py, chaos.py, orchestrator.py, static/)
│  ├─ ledger/     (app.py, chaos.py, repository.py)
│  ├─ fraud/      (app.py, chaos.py, scoring.py)
│  ├─ sanctions/  (app.py, chaos.py, repository.py)
│  ├─ insights/   (app.py, chaos.py, analytics.py)
│  ├─ notifier/   (app.py, chaos.py, consumer.py)
│  └─ audit/      (worker.py, consumer.py)
└─ docs/
   └─ PATHOLOGIES.md
```

`docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use
`depends_on` with `condition: service_healthy` for postgres and redis. Mount the
source read-write and run uvicorn with `--reload` so iteration is fast.

Reserve — but leave empty and unreferenced — a top-level `observability/` directory
with a `.gitkeep`. Later phases fill it.

---

## 12. Makefile targets

| Target | Effect |
|---|---|
| `make up` / `make down` | compose up -d / down |
| `make logs` | `docker compose logs -f` |
| `make seed` | apply `db/schema.sql`, run `scripts/seed.py` |
| `make reset` | `down -v`, `up`, `seed` |
| `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s |
| `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile |
| `make fix-index` | apply `db/optional_indexes.sql` |
| `make drop-index` | drop them again |

---

## 13. Acceptance criteria

The task is done when all of these hold on a clean checkout:

1. `make reset` completes in under 3 minutes and prints the seed summary table.
2. `make smoke` passes with `chaos-clean`.
3. The frontend loads at `http://localhost:8000`, the account dropdown is populated,
   and a submitted transaction returns a settled or rejected result.
4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under
   ~250 ms and zero errors.
5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error
   rate between 2% and 10% — degraded but not fully broken.
6. Toggling any single flag via the chaos panel changes behaviour **without a
   restart**.
7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`.
8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns
   nothing.
9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies.

In your final message, report only: services started, row counts seeded, the
p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you
deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.