LedgerLine/docs/phase-0-base-project-spec.md

Phase 0 — Technical Spec: "Ledgerline" base project

How to use this file: paste the whole thing into your coding agent as the task spec. It is written to be executed without further clarification. Read §2 (Non-goals) yourself before you start — the deliberate omissions are the point of the exercise.

---

1. Mission

Build Ledgerline, a deliberately imperfect microservice system that simulates a small payments/finance backend. It exists for one reason: to be a realistic subject for observability work in later phases. The business logic is fake and does not need to be correct, defensible, or secure. What it does need to be is structurally realistic — real network hops, real database queries, real async message passing, real contention, real latency distributions, and real failure modes.

The system must be runnable with make up && make seed && make smoke on a laptop.

---

2. Non-goals — read carefully

These are hard constraints, not preferences. Violating them destroys the value of the project.

• DO NOT add any observability tooling. No OpenTelemetry, no Prometheus client, no /metrics endpoint, no structured/JSON logging, no trace IDs, no spans, no APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
• DO NOT fix the deliberate pathologies in §7. They are features. Do not add indexes, connection-pool tuning, batching, backoff, or caching beyond what is specified.
• DO NOT add authentication, authorization, rate limiting, or input hardening. A single hardcoded demo user is fine.
• DO NOT write unit tests beyond a single smoke test. Test coverage is not the deliverable.
• DO NOT use Kubernetes, Helm, or any orchestrator. Docker Compose only.
• DO NOT abstract prematurely. Duplication across services is acceptable and preferred over a clever shared framework.

The only logging permitted is Python's standard logging module with a plain-text formatter (§9).

---

3. Technology

Concern	Choice
Language	Python 3.12
Web framework	FastAPI + uvicorn (one worker per service)
HTTP client	httpx (async)
DB	PostgreSQL 16, accessed via SQLAlchemy 2.0 async + asyncpg
Cache / queue	Redis 7 (redis-py asyncio client)
Async messaging	Redis Streams (not Celery, not RabbitMQ)
Validation	Pydantic v2
Packaging	One shared Dockerfile, one image, CMD differs per service
Orchestration	Docker Compose
Schema	A single idempotent db/schema.sql — no Alembic
Frontend	Plain HTML + CSS + vanilla JS, no build step, no framework

Redis Streams is chosen deliberately: it has no automatic context propagation, so a later phase has to implement it manually. Do not swap it for something with built-in instrumentation.

---

4. Services

Seven services (six HTTP, one worker-only). All listen on 0.0.0.0.

#	Service	Port	Role	Stores
1	gateway	8000	Public API + serves the frontend + orchestrates everything	Redis (flags)
2	ledger	8001	Accounts, balances, transaction write path	Postgres
3	fraud	8002	Risk scoring; calls sanctions	Redis (score cache)
4	sanctions	8003	Fake third-party watchlist lookup — the "slow external vendor"	Postgres (read-only)
5	insights	8004	Analytics over transaction history	Postgres (read-heavy) + Redis
6	notifier	8005	HTTP API and a background Redis Stream consumer	Postgres
7	audit	—	Worker only; second consumer group on the same stream	Postgres

Every HTTP service exposes:

• GET /healthz → {"status": "ok", "service": "<name>"} — liveness, no dependency checks
• GET /readyz → checks its own DB/Redis connection, returns 503 if unavailable
• GET / → {"service": "<name>", "version": "0.1.0"}

---

5. Domain model (db/schema.sql)

accounts            (id UUID pk, holder_name text, currency char(3),
                     balance_minor bigint, is_hot boolean default false,
                     created_at timestamptz)

merchant_categories (id smallint pk, code text unique, label text,
                     risk_weight numeric(3,2))

transactions        (id UUID pk, account_id UUID, kind text,          -- deposit|withdrawal|purchase
                     amount_minor bigint, currency char(3),
                     merchant_name text, category_id smallint,
                     status text,                                     -- pending|settled|rejected
                     risk_score numeric(4,3), created_at timestamptz)

sanctions_entries   (id serial pk, normalized_name text, country char(2),
                     list_source text, added_at timestamptz)

notifications       (id UUID pk, transaction_id UUID, channel text,   -- email|push|sms
                     status text, attempts int, sent_at timestamptz)

audit_events        (id bigserial pk, transaction_id UUID, event_type text,
                     payload jsonb, recorded_at timestamptz)

Index policy — important. Create only these indexes:

• primary keys (implicit)
• accounts(holder_name)
• sanctions_entries(normalized_name)

Deliberately omit transactions(account_id, created_at) and transactions(category_id). Put the CREATE INDEX statements for those in db/optional_indexes.sql, applied only by make fix-index.

---

6. Request flows

6.1 POST /api/transactions — the primary flow

The trace this produces should be ~4 levels deep with one async fork.

browser
  └─ gateway  POST /api/transactions
       ├─ ledger   POST /internal/transactions          (create as `pending`, row lock on account)
       ├─ fraud    POST /internal/score                 (risk scoring)
       │     └─ sanctions  POST /internal/lookup        (watchlist check — slow vendor)
       ├─ ledger   POST /internal/transactions/{id}/settle   (or /reject if risk_score > 0.8)
       ├─ redis    XADD  stream:transactions            (fire-and-forget)
       │              ├─ notifier  consumer group `notifier`
       │              └─ audit     consumer group `audit`
       └─ insights GET  /internal/accounts/{id}/summary (cached)

Response body:

{
  "transaction_id": "…", "status": "settled", "risk_score": 0.12,
  "balance_minor": 481200, "summary": { "…": "…" },
  "request_id": "…", "elapsed_ms": 843
}

6.2 GET /api/accounts/{id}/insights — the analytics flow

gateway → insights. insights runs a 120-day aggregation over transactions. This is where the N+1 and the missing index bite.

6.3 GET /api/accounts — list demo accounts for the frontend dropdown

6.4 Admin

• GET /api/admin/chaos → current flag values
• POST /api/admin/chaos → {"flag": "slow_sanctions", "enabled": true}
• POST /api/admin/chaos/profile → {"profile": "clean" | "light" | "nightmare" | "random"}

Flags live in a Redis hash chaos:flags so every service reads the same state without a restart. Each service reads the hash on every request (a tiny bit of extra Redis load is fine and realistic).

---

7. The pathology catalogue

Each pathology lives in a clearly named chaos.py module inside its owning service, with a docstring naming it. All flags default off.

Flag	Service	Behaviour when enabled
slow_sanctions	sanctions	Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts).
sanctions_errors	sanctions	Return HTTP 503 for 3% of requests, chosen at random.
n_plus_one	insights	Instead of a single JOIN on merchant_categories, loop over result rows and issue one SELECT per row.
pool_starvation	ledger	Build the SQLAlchemy engine with pool_size=2, max_overflow=0, pool_timeout=10. (Default otherwise: pool_size=20.)
row_lock_contention	ledger	Wrap the balance update in SELECT … FOR UPDATE and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with is_hot = true.
blocking_event_loop	fraud	Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the async def handler, with no run_in_executor / to_thread.
retry_storm	gateway	On any non-2xx from fraud, retry 3 times immediately — no backoff, no jitter, no circuit breaker.
no_batching	gateway	Split the transaction into 1–5 synthetic "line items" and make one separate fraud call per item, sequentially, instead of one call.
unbounded_cache	insights	Memoize summaries in a module-level dict keyed by (account_id, timestamp_ms) — effectively never hits, never evicts, grows forever.
slow_consumer	notifier	Sleep 250 ms per stream message before acking, so the consumer group lag grows under load.
cold_cache	insights	Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes.
poison_data	ledger	Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them.
chatty_health	gateway	On every /api/transactions call, first GET /healthz on all five downstreams sequentially before doing any real work.

Profiles

• clean — all off.
• light — slow_sanctions, cold_cache.
• nightmare — all thirteen on.
• random — pick a random subset of size 3–6, using a seed derived from time.time() and not logged anywhere. Store the active set in Redis under chaos:flags only, never write it to a file or print it. (This makes the later bottleneck-hunting exercise a genuine hunt.)

Write docs/PATHOLOGIES.md documenting each pathology, its mechanism, and the symptom a reader should expect. Do not reproduce or summarise its contents in your final chat response.

---

8. Seed data (scripts/seed.py)

Deterministic given SEED=42 (settable via env). Idempotent: truncate then insert.

• 800 accounts. Realistic-looking names. Currencies weighted 70% EUR, 20% USD, 10% GBP. Balances log-normal, median ≈ 250 000 minor units. Exactly 5 accounts flagged is_hot = true — these are the contention targets.
• 45 merchant categories with risk_weight between 0.01 and 0.95.
• 250 000 transactions spread over the last 120 days.
  • Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
  • Amounts log-normal.
  • Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest on a Zipf-ish distribution.
  • status 92% settled, 5% rejected, 3% pending.
• 12 000 sanctions_entries. ~40 of them must match names of seeded account holders, so a meaningful fraction of requests genuinely score as high risk.
• ~2% malformed rows distributed through transactions:
  • currency = NULL
  • negative amount_minor on a deposit
  • merchant_name containing 4-byte emoji and a NUL-ish escape sequence
  • created_at set 3 days in the future
  • category_id pointing at a non-existent category

Print a summary table at the end (row counts per table, hot account IDs, count of malformed rows). Seeding 250k rows must complete in under 60 seconds — use COPY / executemany, not per-row inserts.

---

9. Logging (the deliberately bad baseline)

Every service configures:

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
)

• Log lines are plain text, human-prose style: logger.info(f"processing tx for account {account_id}, amount {amount}"). Not JSON. Not key-value.
• gateway generates an X-Request-Id (uuid4) per inbound request and forwards it as a header on every downstream httpx call. Each service logs it manually at the start of each handler. This is the only correlation mechanism that exists. It is intentionally weak — it does not survive the Redis Stream hop, it carries no timing, and it cannot express parent/child relationships.
• Set PYTHONUNBUFFERED=1 in the Dockerfile.
• Log exceptions with logger.exception(...), but do not add any error-tracking SDK.

---

10. Frontend

Served by gateway at / from gateway/static/. Three files: index.html, app.js, style.css. No bundler, no npm, no CDN dependencies.

Layout, top to bottom:

1. Submit a transaction — account dropdown (populated from GET /api/accounts), amount input, merchant name input, kind radio (deposit / withdrawal / purchase), Submit button.
2. Result panel — pretty-printed JSON response, plus the browser-measured round-trip time in ms, plus the request_id shown in a monospace font that is easy to copy.
3. Load burst — a button that fires N concurrent requests (N selectable: 10 / 50 / 200) against random accounts, then renders a small table: count, success, errors, min / mean / p50 / p95 / max ms, measured client-side. This is a crude stand-in for a real load tool; a later phase replaces it with k6.
4. Chaos panel — a checkbox per flag reflecting current state from GET /api/admin/chaos, toggling via POST /api/admin/chaos, plus four profile buttons (clean, light, nightmare, random).
5. Service health strip — polls /healthz of all six HTTP services every 5 s via a GET /api/health/all aggregation endpoint on the gateway, showing a green or red dot per service.

Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.

---

11. Repository layout

ledgerline/
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile                  # single image for all Python services
├─ pyproject.toml              # or requirements.txt — one dependency set
├─ .env.example
├─ README.md
├─ db/
│  ├─ schema.sql
│  └─ optional_indexes.sql
├─ scripts/
│  ├─ seed.py
│  └─ smoke.py
├─ common/
│  ├─ __init__.py
│  ├─ settings.py              # pydantic-settings, env-driven
│  ├─ db.py                    # async engine/session factory
│  ├─ redis_client.py
│  ├─ http.py                  # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
│  └─ logging_config.py        # the plain-text logging setup from §9
├─ services/
│  ├─ gateway/    (app.py, chaos.py, orchestrator.py, static/)
│  ├─ ledger/     (app.py, chaos.py, repository.py)
│  ├─ fraud/      (app.py, chaos.py, scoring.py)
│  ├─ sanctions/  (app.py, chaos.py, repository.py)
│  ├─ insights/   (app.py, chaos.py, analytics.py)
│  ├─ notifier/   (app.py, chaos.py, consumer.py)
│  └─ audit/      (worker.py, consumer.py)
└─ docs/
   └─ PATHOLOGIES.md

docker-compose.yml defines: postgres, redis, and the seven services. Use depends_on with condition: service_healthy for postgres and redis. Mount the source read-write and run uvicorn with --reload so iteration is fast.

Reserve — but leave empty and unreferenced — a top-level observability/ directory with a .gitkeep. Later phases fill it.

---

12. Makefile targets

Target	Effect
make up / make down	compose up -d / down
make logs	docker compose logs -f
make seed	apply db/schema.sql, run scripts/seed.py
make reset	down -v, up, seed
make smoke	run scripts/smoke.py — one transaction, one insights call, assert 200s
make chaos-clean / chaos-light / chaos-nightmare / chaos-random	set the profile
make fix-index	apply db/optional_indexes.sql
make drop-index	drop them again

---

13. Acceptance criteria

The task is done when all of these hold on a clean checkout:

1. make reset completes in under 3 minutes and prints the seed summary table.
2. make smoke passes with chaos-clean.
3. The frontend loads at http://localhost:8000, the account dropdown is populated, and a submitted transaction returns a settled or rejected result.
4. With chaos-clean, a 200-request burst from the frontend shows p95 under ~250 ms and zero errors.
5. With chaos-nightmare, the same burst shows p95 at least 5× worse and an error rate between 2% and 10% — degraded but not fully broken.
6. Toggling any single flag via the chaos panel changes behaviour without a restart.
7. docker compose logs gateway shows plain-text lines carrying X-Request-Id.
8. grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" . returns nothing.
9. docs/PATHOLOGIES.md exists and documents all thirteen pathologies.

In your final message, report only: services started, row counts seeded, the p95/error-rate numbers you measured for clean vs nightmare, and anything you deviated from in this spec. Do not summarise docs/PATHOLOGIES.md.