# Ledgerline A deliberately imperfect seven-service payments backend, built as a subject for learning observability with OpenTelemetry and the Grafana stack. > ### ⚠️ This system is broken on purpose > > Ledgerline ships with thirteen intentional performance and reliability > pathologies: a missing index, a starved connection pool, an N+1 query, a > blocking call inside an `async def`, a retry storm with no backoff, an > unbounded cache, and more. They are toggled at runtime and documented in > [`docs/PATHOLOGIES.md`](docs/PATHOLOGIES.md). > > **Do not fix them.** They are the entire point. If you are here to contribute, > read [`CLAUDE.md`](CLAUDE.md) first. The business logic is fake and does not need to be correct, secure, or defensible. What it needs to be is *structurally* realistic: real network hops, real SQL, real async message passing, real contention, and real latency distributions — so that instrumenting it teaches something true. --- ## Quickstart Requires Docker and Docker Compose. Nothing else. ```bash git clone && cd ledgerline cp .env.example .env make up # start postgres, redis, and the seven services make seed # schema + 250k transactions across 800 accounts (~60s) make smoke # end-to-end sanity check make test-contract # fast functional suite (~60s) ``` Then open **http://localhost:8000**. The observability backend lives in a separate Ansible repo and is deployed to its own host. From Phase 2 onward: ```bash # in observability-ansible/ ansible-playbook playbooks/grafana-stack.yml ``` Then point Ledgerline at it via `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`. Reset everything, including volumes: ```bash make reset ``` --- ## Architecture ```mermaid flowchart LR B[browser] --> GW[gateway :8000] GW --> LD[ledger :8001] GW --> FR[fraud :8002] FR --> SA[sanctions :8003] GW --> IN[insights :8004] GW -- XADD --> ST[(redis stream)] ST --> NO[notifier :8005] ST --> AU[audit worker] LD --> PG[(postgres)] SA --> PG IN --> PG NO --> PG AU --> PG IN --> RD[(redis cache)] ``` | Service | Port | Role | Stores | |---|---|---|---| | `gateway` | 8000 | Public API, serves the frontend, orchestrates everything | Redis (flags) | | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres | | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) | | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow vendor" | Postgres (read) | | `insights` | 8004 | 120-day analytics over transaction history | Postgres + Redis | | `notifier` | 8005 | HTTP API **and** a Redis Stream consumer | Postgres | | `audit` | — | Worker only; second consumer group on the same stream | Postgres | Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz` (dependency check). **Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no automatic trace-context propagation, so Phase 3 has to implement it by hand. That's the most valuable exercise in the project. Don't swap it out. ### The primary request flow `POST /api/transactions` produces a trace roughly four levels deep with one async fork: ``` gateway ├─ ledger create transaction (pending, row lock on account) ├─ fraud score │ └─ sanctions watchlist lookup ├─ ledger settle or reject ├─ redis XADD stream:transactions ──┬─ notifier (consumer group) │ └─ audit (consumer group) └─ insights cached account summary ``` The second flow, `GET /api/accounts/{id}/insights`, is a read-heavy aggregation and behaves very differently under load. --- ## Chaos flags Pathologies are controlled at runtime via a Redis hash (`chaos:flags`), read on every request — **no restart needed**. Toggle them from the frontend's chaos panel, the API, or the Makefile. ```bash make chaos-clean # all off — the baseline make chaos-light # a mild, realistic amount of pain make chaos-nightmare # all thirteen on make chaos-random # 3–6 flags, seed never logged ``` ```bash curl -s localhost:8000/api/admin/chaos | jq curl -sX POST localhost:8000/api/admin/chaos \ -H 'content-type: application/json' \ -d '{"flag":"slow_sanctions","enabled":true}' ``` Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`, `pool_starvation`, `row_lock_contention`, `blocking_event_loop`, `retry_storm`, `no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`, `chatty_health`. > **During Phase 9, don't open the chaos panel and don't call > `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the > seed; the point is to find the pathologies from telemetry alone. The panel > will happily tell you the answer, which is the one thing you don't want. --- ## Make targets | Target | Effect | |---|---| | `up` / `down` | Start / stop the application stack | | `logs` | `docker compose logs -f` | | `seed` | Apply schema and seed 250k transactions | | `reset` | `down -v` → `up` → `seed` | | `smoke` | One transaction + one insights call, assert 200s | | `test` | Full suite except the slow markers | | `test-contract` | Functional + seed + guard assertions, < 60 s | | `test-baseline` | Is the clean system genuinely healthy | | `test-pathology` | One differential test per pathology — ⚠️ answer key, see below | | `test-slow` | The soak-ish pathology tests | | `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile | | `fix-index` / `drop-index` | Add or remove the deliberately missing indexes | | `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 6+* | | `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 7+* | --- ## Testing Phase 0 was built **test-first, by two separate agents**: one wrote `tests/` from `docs/phase-0a-test-suite-spec.md` without seeing any implementation, the second implemented the services until the suite went green without being allowed to touch a test. The suite is **black-box**. It imports nothing from the application and talks to the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals, not mechanics — there are no references to function names, module paths, or SQL strings anywhere in it. Two halves: - **Working state.** A transaction really traverses all seven services (stop `sanctions` and transactions must fail — that's the test that catches a collapsed monolith); async work really happens off the request path via two independent consumer groups; the seed data is really 250k rows, really skewed toward five hot accounts, and really contains all five varieties of poison row. - **Broken state.** One differential test per pathology, asserting the system measurably degrades when a flag is on versus off. Assertions favour signals that are hard to fake: `pg_stat_statements` query counts for the N+1, `EXPLAIN` plan node types for the missing index, Redis consumer-group lag for the slow consumer, container RSS for the memory leak, throughput plateaus for pool starvation. A `sleep()` won't satisfy those. ```bash make up && make seed # prerequisite — the suite runs against a live stack make test-contract # fast make test # everything except slow ``` > ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so > its output is a complete answer key. Don't run it during the Phase 9 > bottleneck hunt. `tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no `prometheus_client`, no structured logging, no `/metrics`. **Delete it at the start of Phase 3**; it has done its job by then. Some pathology assertions currently count log lines, because plain text is the only cross-service observability surface that exists at Phase 0. That is the deficiency this project sets out to fix — those assertions get rewritten against structured logs in Phase 4 and span data in Phase 5, and become both simpler and more robust. See `tests/README.md`. --- ## Ports | | Port | | |---|---|---| | Frontend / gateway | 8000 | | | `ledger` … `notifier` | 8001–8005 | | | Postgres | 5432 | `ledgerline` / `ledgerline` | | Redis | 6379 | | On the observability host (deployed by `observability-ansible`): | | Port | | |---|---|---| | Grafana | 3000 | credentials from `group_vars/all.yml` | | Prometheus | 9090 | remote-write receiver enabled | | Alertmanager | 9093 | | | Loki | 3100 | | | Tempo | 3200 | | | Alloy UI | 12345 | component graph — check here first when data is missing | | OTLP (Alloy) | 4317 / 4318 | gRPC / HTTP | | Node Exporter | 9100 | | --- ## Observability stack The backend is **not** part of this repo. It is deployed by a separate Ansible project, `observability-ansible`, onto a target host: ``` Ledgerline /metrics ──────scrape──────────────┐ container stdout ──docker.sock──▶ Alloy ──▶ Loki Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo │ span metrics + exemplars ▼ (remote_write) Prometheus ◀────────┘ │ Alertmanager ──▶ Receivers │ Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above Correlation key: trace_id / exemplars ``` RED metrics and the service graph are generated by **Tempo's metrics-generator**, not by collector connectors — do not add `spanmetrics`/`servicegraph` to Alloy. Three constraints inherited from that platform, worth knowing before you debug anything: - **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are rejected with a 404 on `/v1/metrics` and `/v1/logs`, and the only sign is an export error in the sending app's logs. This is by design; the platform's diagram labels that input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`. - **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`, case-sensitively.** Log lines must carry lowercase `trace_id`. OTel's logging auto-instrumentation emits `otelTraceID` and will not match. - **The platform provisions datasources only.** Every dashboard here is generated from code with the Grafana Foundation SDK and shipped through the Ansible role, so `ansible-playbook` rebuilds them from scratch. **Never save a dashboard from the Grafana UI** — it won't survive the next run. Set `OTEL_EXPORTER_OTLP_ENDPOINT` (and friends) in `.env` to point at the Alloy host. Nothing observability-related is hardcoded. --- ## The five-minute demo Rehearse this before showing anyone. 1. Delete every dashboard in the Grafana UI, then re-run `ansible-playbook playbooks/grafana-stack.yml` — platform and dashboards both rebuilt from code, one command. 2. Submit one transaction from the frontend. Follow that single request through Grafana: RED dashboard → latency spike → exemplar → trace → the slow span → that span's logs → back to the trace. One request, three signals, no searching. 3. `make chaos-nightmare && make load-stress` — watch p95 diverge from p50, the service map turn red at a specific node, the error budget burn, and the fast-burn alert fire. 4. Walk through [`docs/INVESTIGATION.md`](docs/INVESTIGATION.md) — how the pathologies were found from telemetry alone. --- ## Repository layout ``` ├─ CLAUDE.md # instructions for AI assistants — read before contributing ├─ Makefile ├─ docker-compose.yml ├─ Dockerfile # one image, all Python services ├─ db/ │ ├─ schema.sql │ ├─ optional_indexes.sql # the deliberately-omitted indexes │ └─ seed_manifest.json # written by the seeder; the tests' source of truth ├─ scripts/ # seed.py, smoke.py ├─ common/ # settings, db, redis, http, logging, telemetry ├─ services/ # gateway ledger fraud sanctions insights notifier audit │ └─ */chaos.py # each service's pathologies, clearly marked ├─ tests/ # black-box integration suite — a frozen contract │ ├─ harness/ # load driver, pg_stat helpers, stream + container probes │ └─ README.md # methodology, markers, known-flaky list ├─ observability/ # dashboard generator (Foundation SDK); output ships via the Ansible role ├─ loadtest/ # k6 scripts └─ docs/ ├─ roadmap/ # the ten-phase learning plan, one file per phase │ ├─ intro.md # platform background, resource spine (read first) │ ├─ phase-0.md … phase-11.md # one phase per file — status, message, concepts, build │ └─ summary.md # one-page status/sequencing overview of all phases ├─ phase-0a-test-suite-spec.md # spec the test suite was written from ├─ phase-0-base-project-spec.md # spec the base app was built from ├─ PATHOLOGIES.md # ⚠️ spoilers — the answer key ├─ SLO.md # SLIs, SLOs, SLA, RTO/RPO, error budget policy ├─ CONCEPTS.md # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words ├─ INVESTIGATION.md # the Phase 9 bottleneck hunt, written as an incident doc └─ PIPELINE-CHOICES.md # Filebeat vs Logstash vs OpenTelemetry — when and why ``` --- ## Status - [ ] **Phase 0a** — integration test suite, written first, red against nothing - [ ] **Phase 0b** — base project implemented until the suite is green - [x] **Phase 1** — deployment: Docker Compose + Traefik on a server - [ ] **Phase 2** — concepts, first trace end to end - [ ] **Phase 3** — distributed tracing across all seven services - [ ] **Phase 4** — structured logs, correlated with traces - [ ] **Phase 5** — metrics, RED, service map, exemplars - [ ] **Phase 6** — dashboards and alerts as code - [ ] **Phase 7** — k6 load testing, percentiles - [ ] **Phase 8** — SLIs, SLOs, error budgets, RTO/RPO - [ ] **Phase 9** — bottleneck hunt - [ ] **Phase 10** — Filebeat / Logstash / OpenTelemetry judgment - [ ] **Phase 11** — delivery Full plan: [`docs/roadmap/`](docs/roadmap/) — start with [`intro.md`](docs/roadmap/intro.md), then the phase you're on, or [`summary.md`](docs/roadmap/summary.md) for the one-page overview. --- ## Non-goals No authentication, no authorization, no input hardening, no unit tests, no Kubernetes, no production-grade storage. Ledgerline runs on a laptop under Docker Compose and is never exposed to a network you don't control. The integration suite is deliberately the *only* test layer: it exists to pin down the system's observable behaviour — working and broken — not to verify the fake business logic. Things deliberately left for a later iteration, and worth naming when discussing the project: tail-based sampling instead of 100%, Mimir instead of raw Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII redaction in the Collector, and multi-tenancy.