mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add project documentation and Claude instructions

Browse Source 
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.
This commit is contained in:
Movses Aslanyan
2026-09-05 08:11:00 +03:30
commit eb99cdbb8c
19 changed files with 3372 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-0-base-project-spec.md
+373
View File
@@ -0,0 +1,373 @@
# Phase 0 — Technical Spec: "Ledgerline" base project
> **How to use this file:** paste the whole thing into your coding agent as the task
> spec. It is written to be executed without further clarification. Read
> §2 (Non-goals) yourself before you start — the deliberate omissions are the
> point of the exercise.
---
## 1. Mission
Build **Ledgerline**, a deliberately imperfect microservice system that simulates a
small payments/finance backend. It exists for one reason: to be a realistic subject
for observability work in later phases. The business logic is fake and does not
need to be correct, defensible, or secure. What it *does* need to be is
**structurally realistic** — real network hops, real database queries, real async
message passing, real contention, real latency distributions, and real failure modes.
The system must be runnable with `make up && make seed && make smoke` on a laptop.
---
## 2. Non-goals — read carefully
These are **hard constraints**, not preferences. Violating them destroys the value
of the project.
- **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client,
no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no
APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
- **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add
indexes, connection-pool tuning, batching, backoff, or caching beyond what is
specified.
- **DO NOT add authentication, authorization, rate limiting, or input hardening.**
A single hardcoded demo user is fine.
- **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the
deliverable.
- **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only.
- **DO NOT abstract prematurely.** Duplication across services is acceptable and
preferred over a clever shared framework.
The only logging permitted is Python's standard `logging` module with a plain-text
formatter (§9).
---
## 3. Technology
| Concern | Choice |
|---|---|
| Language | Python 3.12 |
| Web framework | FastAPI + `uvicorn` (one worker per service) |
| HTTP client | `httpx` (async) |
| DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` |
| Cache / queue | Redis 7 (`redis-py` asyncio client) |
| Async messaging | Redis **Streams** (not Celery, not RabbitMQ) |
| Validation | Pydantic v2 |
| Packaging | One shared `Dockerfile`, one image, `CMD` differs per service |
| Orchestration | Docker Compose |
| Schema | A single idempotent `db/schema.sql` — no Alembic |
| Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework |
Redis Streams is chosen deliberately: it has **no automatic context propagation**,
so a later phase has to implement it manually. Do not swap it for something with
built-in instrumentation.
---
## 4. Services
Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`.
| # | Service | Port | Role | Stores |
|---|---|---|---|---|
| 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) |
| 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
| 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
| 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) |
| 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis |
| 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres |
| 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |
Every HTTP service exposes:
- `GET /healthz` → `{"status": "ok", "service": "<name>"}` — liveness, no dependency checks
- `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable
- `GET /` → `{"service": "<name>", "version": "0.1.0"}`
---
## 5. Domain model (`db/schema.sql`)
```
accounts (id UUID pk, holder_name text, currency char(3),
balance_minor bigint, is_hot boolean default false,
created_at timestamptz)
merchant_categories (id smallint pk, code text unique, label text,
risk_weight numeric(3,2))
transactions (id UUID pk, account_id UUID, kind text, -- deposit|withdrawal|purchase
amount_minor bigint, currency char(3),
merchant_name text, category_id smallint,
status text, -- pending|settled|rejected
risk_score numeric(4,3), created_at timestamptz)
sanctions_entries (id serial pk, normalized_name text, country char(2),
list_source text, added_at timestamptz)
notifications (id UUID pk, transaction_id UUID, channel text, -- email|push|sms
status text, attempts int, sent_at timestamptz)
audit_events (id bigserial pk, transaction_id UUID, event_type text,
payload jsonb, recorded_at timestamptz)
```
**Index policy — important.** Create only these indexes:
- primary keys (implicit)
- `accounts(holder_name)`
- `sanctions_entries(normalized_name)`
Deliberately **omit** `transactions(account_id, created_at)` and
`transactions(category_id)`. Put the `CREATE INDEX` statements for those in
`db/optional_indexes.sql`, applied only by `make fix-index`.
---
## 6. Request flows
### 6.1 `POST /api/transactions` — the primary flow
The trace this produces should be ~4 levels deep with one async fork.
```
browser
└─ gateway POST /api/transactions
├─ ledger POST /internal/transactions (create as `pending`, row lock on account)
├─ fraud POST /internal/score (risk scoring)
│ └─ sanctions POST /internal/lookup (watchlist check — slow vendor)
├─ ledger POST /internal/transactions/{id}/settle (or /reject if risk_score > 0.8)
├─ redis XADD stream:transactions (fire-and-forget)
│ ├─ notifier consumer group `notifier`
│ └─ audit consumer group `audit`
└─ insights GET /internal/accounts/{id}/summary (cached)
```
Response body:
```json
{
"transaction_id": "…", "status": "settled", "risk_score": 0.12,
"balance_minor": 481200, "summary": { "…": "…" },
"request_id": "…", "elapsed_ms": 843
}
```
### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow
`gateway → insights`. `insights` runs a 120-day aggregation over `transactions`.
This is where the N+1 and the missing index bite.
### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown
### 6.4 Admin
- `GET /api/admin/chaos` → current flag values
- `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}`
- `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}`
Flags live in a Redis hash `chaos:flags` so every service reads the same state
without a restart. Each service reads the hash on **every request** (a tiny bit of
extra Redis load is fine and realistic).
---
## 7. The pathology catalogue
Each pathology lives in a clearly named `chaos.py` module inside its owning service,
with a docstring naming it. All flags default **off**.
| Flag | Service | Behaviour when enabled |
|---|---|---|
| `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). |
| `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. |
| `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. |
| `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) |
| `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. |
| `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. |
| `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. |
| `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. |
| `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. |
| `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. |
| `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. |
| `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. |
| `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. |
**Profiles**
- `clean` — all off.
- `light` — `slow_sanctions`, `cold_cache`.
- `nightmare` — all thirteen on.
- `random` — pick a random subset of size 3–6, using a seed derived from
`time.time()` and **not logged anywhere**. Store the active set in Redis under
`chaos:flags` only, never write it to a file or print it. (This makes the later
bottleneck-hunting exercise a genuine hunt.)
Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the
symptom a reader should expect. Do **not** reproduce or summarise its contents in
your final chat response.
---
## 8. Seed data (`scripts/seed.py`)
Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert.
- **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD,
10% GBP. Balances log-normal, median ≈ 250 000 minor units.
Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets.
- **45 merchant categories** with `risk_weight` between 0.01 and 0.95.
- **250 000 transactions** spread over the last 120 days.
- Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
- Amounts log-normal.
- Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest
on a Zipf-ish distribution.
- `status` 92% `settled`, 5% `rejected`, 3% `pending`.
- **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account
holders, so a meaningful fraction of requests genuinely score as high risk.
- **~2% malformed rows** distributed through `transactions`:
- `currency` = `NULL`
- negative `amount_minor` on a `deposit`
- `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence
- `created_at` set 3 days in the future
- `category_id` pointing at a non-existent category
Print a summary table at the end (row counts per table, hot account IDs, count of
malformed rows). Seeding 250k rows must complete in under 60 seconds — use
`COPY` / `executemany`, not per-row inserts.
---
## 9. Logging (the deliberately bad baseline)
Every service configures:
```python
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
```
- Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value.
- `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it
as a header on every downstream `httpx` call. Each service logs it manually at the
start of each handler. **This is the only correlation mechanism that exists.**
It is intentionally weak — it does not survive the Redis Stream hop, it carries no
timing, and it cannot express parent/child relationships.
- Set `PYTHONUNBUFFERED=1` in the Dockerfile.
- Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK.
---
## 10. Frontend
Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`,
`app.js`, `style.css`. No bundler, no npm, no CDN dependencies.
Layout, top to bottom:
1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`),
amount input, merchant name input, kind radio (deposit / withdrawal / purchase),
Submit button.
2. **Result panel** — pretty-printed JSON response, plus the browser-measured
round-trip time in ms, plus the `request_id` shown in a monospace font that is
easy to copy.
3. **Load burst** — a button that fires *N* concurrent requests (N selectable:
10 / 50 / 200) against random accounts, then renders a small table:
count, success, errors, min / mean / p50 / p95 / max ms, measured client-side.
This is a crude stand-in for a real load tool; a later phase replaces it with k6.
4. **Chaos panel** — a checkbox per flag reflecting current state from
`GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile
buttons (`clean`, `light`, `nightmare`, `random`).
5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s
via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green
or red dot per service.
Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.
---
## 11. Repository layout
```
ledgerline/
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile # single image for all Python services
├─ pyproject.toml # or requirements.txt — one dependency set
├─ .env.example
├─ README.md
├─ db/
│ ├─ schema.sql
│ └─ optional_indexes.sql
├─ scripts/
│ ├─ seed.py
│ └─ smoke.py
├─ common/
│ ├─ __init__.py
│ ├─ settings.py # pydantic-settings, env-driven
│ ├─ db.py # async engine/session factory
│ ├─ redis_client.py
│ ├─ http.py # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
│ └─ logging_config.py # the plain-text logging setup from §9
├─ services/
│ ├─ gateway/ (app.py, chaos.py, orchestrator.py, static/)
│ ├─ ledger/ (app.py, chaos.py, repository.py)
│ ├─ fraud/ (app.py, chaos.py, scoring.py)
│ ├─ sanctions/ (app.py, chaos.py, repository.py)
│ ├─ insights/ (app.py, chaos.py, analytics.py)
│ ├─ notifier/ (app.py, chaos.py, consumer.py)
│ └─ audit/ (worker.py, consumer.py)
└─ docs/
└─ PATHOLOGIES.md
```
`docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use
`depends_on` with `condition: service_healthy` for postgres and redis. Mount the
source read-write and run uvicorn with `--reload` so iteration is fast.
Reserve — but leave empty and unreferenced — a top-level `observability/` directory
with a `.gitkeep`. Later phases fill it.
---
## 12. Makefile targets
| Target | Effect |
|---|---|
| `make up` / `make down` | compose up -d / down |
| `make logs` | `docker compose logs -f` |
| `make seed` | apply `db/schema.sql`, run `scripts/seed.py` |
| `make reset` | `down -v`, `up`, `seed` |
| `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s |
| `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile |
| `make fix-index` | apply `db/optional_indexes.sql` |
| `make drop-index` | drop them again |
---
## 13. Acceptance criteria
The task is done when all of these hold on a clean checkout:
1. `make reset` completes in under 3 minutes and prints the seed summary table.
2. `make smoke` passes with `chaos-clean`.
3. The frontend loads at `http://localhost:8000`, the account dropdown is populated,
and a submitted transaction returns a settled or rejected result.
4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under
~250 ms and zero errors.
5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error
rate between 2% and 10% — degraded but not fully broken.
6. Toggling any single flag via the chaos panel changes behaviour **without a
restart**.
7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`.
8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns
nothing.
9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies.
In your final message, report only: services started, row counts seeded, the
p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you
deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.