Ledgerline

A deliberately imperfect seven-service payments backend, built as a subject for learning observability with OpenTelemetry and the Grafana stack.

⚠️ This system is broken on purpose

Ledgerline ships with thirteen intentional performance and reliability pathologies: a missing index, a starved connection pool, an N+1 query, a blocking call inside an async def, a retry storm with no backoff, an unbounded cache, and more. They are toggled at runtime and documented in docs/PATHOLOGIES.md.

Do not fix them. They are the entire point. If you are here to contribute, read CLAUDE.md first.

The business logic is fake and does not need to be correct, secure, or defensible. What it needs to be is structurally realistic: real network hops, real SQL, real async message passing, real contention, and real latency distributions — so that instrumenting it teaches something true.

---

Quickstart

Requires Docker and Docker Compose. Nothing else.

git clone <this-repo> && cd ledgerline
cp .env.example .env
make up             # start postgres, redis, and the seven services
make seed           # schema + 250k transactions across 800 accounts (~60s)
make smoke          # end-to-end sanity check
make test-contract  # fast functional suite (~60s)

Then open http://localhost:8000.

The observability backend lives in a separate Ansible repo and is deployed to its own host. From Phase 2 onward:

# in observability-ansible/
ansible-playbook playbooks/grafana-stack.yml

Then point Ledgerline at it via OTEL_EXPORTER_OTLP_ENDPOINT in .env.

Reset everything, including volumes:

make reset

---

Architecture

flowchart LR
    B[browser] --> GW[gateway :8000]
    GW --> LD[ledger :8001]
    GW --> FR[fraud :8002]
    FR --> SA[sanctions :8003]
    GW --> IN[insights :8004]
    GW -- XADD --> ST[(redis stream)]
    ST --> NO[notifier :8005]
    ST --> AU[audit worker]
    LD --> PG[(postgres)]
    SA --> PG
    IN --> PG
    NO --> PG
    AU --> PG
    IN --> RD[(redis cache)]

Service	Port	Role	Stores
gateway	8000	Public API, serves the frontend, orchestrates everything	Redis (flags)
ledger	8001	Accounts, balances, transaction write path	Postgres
fraud	8002	Risk scoring; calls sanctions	Redis (score cache)
sanctions	8003	Fake third-party watchlist lookup — the "slow vendor"	Postgres (read)
insights	8004	120-day analytics over transaction history	Postgres + Redis
notifier	8005	HTTP API and a Redis Stream consumer	Postgres
audit	—	Worker only; second consumer group on the same stream	Postgres

Every HTTP service exposes GET /healthz (liveness) and GET /readyz (dependency check).

Redis Streams is used deliberately rather than Celery or RabbitMQ: it has no automatic trace-context propagation, so Phase 3 has to implement it by hand. That's the most valuable exercise in the project. Don't swap it out.

The primary request flow

POST /api/transactions produces a trace roughly four levels deep with one async fork:

gateway
  ├─ ledger    create transaction (pending, row lock on account)
  ├─ fraud     score
  │    └─ sanctions   watchlist lookup
  ├─ ledger    settle or reject
  ├─ redis     XADD stream:transactions  ──┬─ notifier  (consumer group)
  │                                        └─ audit     (consumer group)
  └─ insights  cached account summary

The second flow, GET /api/accounts/{id}/insights, is a read-heavy aggregation and behaves very differently under load.

---

Chaos flags

Pathologies are controlled at runtime via a Redis hash (chaos:flags), read on every request — no restart needed. Toggle them from the frontend's chaos panel, the API, or the Makefile.

make chaos-clean       # all off — the baseline
make chaos-light       # a mild, realistic amount of pain
make chaos-nightmare   # all thirteen on
make chaos-random      # 3–6 flags, seed never logged

curl -s localhost:8000/api/admin/chaos | jq
curl -sX POST localhost:8000/api/admin/chaos \
  -H 'content-type: application/json' \
  -d '{"flag":"slow_sanctions","enabled":true}'

Flag names: slow_sanctions, sanctions_errors, n_plus_one, pool_starvation, row_lock_contention, blocking_event_loop, retry_storm, no_batching, unbounded_cache, slow_consumer, cold_cache, poison_data, chatty_health.

During Phase 9, don't open the chaos panel and don't call GET /api/admin/chaos. make chaos-random picks a subset and never logs the seed; the point is to find the pathologies from telemetry alone. The panel will happily tell you the answer, which is the one thing you don't want.

---

Make targets

Target	Effect
up / down	Start / stop the application stack
logs	docker compose logs -f
seed	Apply schema and seed 250k transactions
reset	down -v → up → seed
smoke	One transaction + one insights call, assert 200s
test	Full suite except the slow markers
test-contract	Functional + seed + guard assertions, < 60 s
test-baseline	Is the clean system genuinely healthy
test-pathology	One differential test per pathology — ⚠️ answer key, see below
test-slow	The soak-ish pathology tests
chaos-clean / -light / -nightmare / -random	Set the chaos profile
fix-index / drop-index	Add or remove the deliberately missing indexes
dashboards	Regenerate dashboard JSON via the Grafana Foundation SDK — Phase 6+
load-smoke / load-average / load-stress / load-soak	k6 profiles — Phase 7+

---

Testing

Phase 0 was built test-first, by two separate agents: one wrote tests/ from docs/phase-0a-test-suite-spec.md without seeing any implementation, the second implemented the services until the suite went green without being allowed to touch a test.

The suite is black-box. It imports nothing from the application and talks to the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals, not mechanics — there are no references to function names, module paths, or SQL strings anywhere in it.

Two halves:

• Working state. A transaction really traverses all seven services (stop sanctions and transactions must fail — that's the test that catches a collapsed monolith); async work really happens off the request path via two independent consumer groups; the seed data is really 250k rows, really skewed toward five hot accounts, and really contains all five varieties of poison row.
• Broken state. One differential test per pathology, asserting the system measurably degrades when a flag is on versus off. Assertions favour signals that are hard to fake: pg_stat_statements query counts for the N+1, EXPLAIN plan node types for the missing index, Redis consumer-group lag for the slow consumer, container RSS for the memory leak, throughput plateaus for pool starvation. A sleep() won't satisfy those.

make up && make seed      # prerequisite — the suite runs against a live stack
make test-contract        # fast
make test                 # everything except slow

⚠️ make test-pathology is a spoiler. It has one named test per flag, so its output is a complete answer key. Don't run it during the Phase 9 bottleneck hunt.

tests/test_no_observability.py guards the Phase 0 "before" state — no OTel, no prometheus_client, no structured logging, no /metrics. Delete it at the start of Phase 3; it has done its job by then.

Some pathology assertions currently count log lines, because plain text is the only cross-service observability surface that exists at Phase 0. That is the deficiency this project sets out to fix — those assertions get rewritten against structured logs in Phase 4 and span data in Phase 5, and become both simpler and more robust. See tests/README.md.

---

Ports

	Port
Frontend / gateway	8000
ledger … notifier	8001–8005
Postgres	5432	ledgerline / ledgerline
Redis	6379

On the observability host (deployed by observability-ansible):

	Port
Grafana	3000	credentials from group_vars/all.yml
Prometheus	9090	remote-write receiver enabled
Alertmanager	9093
Loki	3100
Tempo	3200
Alloy UI	12345	component graph — check here first when data is missing
OTLP (Alloy)	4317 / 4318	gRPC / HTTP
Node Exporter	9100

---

Observability stack

The backend is not part of this repo. It is deployed by a separate Ansible project, observability-ansible, onto a target host:

Ledgerline /metrics ──────scrape──────────────┐
container stdout ──docker.sock──▶ Alloy ──▶ Loki
Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo
                                              │  span metrics + exemplars
                                              ▼         (remote_write)
                                         Prometheus ◀────────┘
                                              │
                                       Alertmanager ──▶ Receivers
                                              │
   Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above
          Correlation key: trace_id / exemplars

RED metrics and the service graph are generated by Tempo's metrics-generator, not by collector connectors — do not add spanmetrics/servicegraph to Alloy.

Three constraints inherited from that platform, worth knowing before you debug anything:

• Alloy's OTLP receiver forwards traces only. OTLP metrics and logs are accepted and silently dropped — by design; the platform's diagram labels that input "spans + resource attrs". Metrics reach Prometheus by direct scrape of each service's /metrics endpoint, registered in prometheus_extra_scrape_jobs.
• Log→trace correlation matches trace[_-]?id["=: ]+([a-fA-F0-9]+), case-sensitively. Log lines must carry lowercase trace_id. OTel's logging auto-instrumentation emits otelTraceID and will not match.
• The platform provisions datasources only. Every dashboard here is generated from code with the Grafana Foundation SDK and shipped through the Ansible role, so ansible-playbook rebuilds them from scratch. Never save a dashboard from the Grafana UI — it won't survive the next run.

Set OTEL_EXPORTER_OTLP_ENDPOINT (and friends) in .env to point at the Alloy host. Nothing observability-related is hardcoded.

---

The five-minute demo

Rehearse this before showing anyone.

1. Delete every dashboard in the Grafana UI, then re-run ansible-playbook playbooks/grafana-stack.yml — platform and dashboards both rebuilt from code, one command.
2. Submit one transaction from the frontend. Follow that single request through Grafana: RED dashboard → latency spike → exemplar → trace → the slow span → that span's logs → back to the trace. One request, three signals, no searching.
3. make chaos-nightmare && make load-stress — watch p95 diverge from p50, the service map turn red at a specific node, the error budget burn, and the fast-burn alert fire.
4. Walk through docs/INVESTIGATION.md — how the pathologies were found from telemetry alone.

---

Repository layout

├─ CLAUDE.md                  # instructions for AI assistants — read before contributing
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile                 # one image, all Python services
├─ db/
│  ├─ schema.sql
│  ├─ optional_indexes.sql    # the deliberately-omitted indexes
│  └─ seed_manifest.json      # written by the seeder; the tests' source of truth
├─ scripts/                   # seed.py, smoke.py
├─ common/                    # settings, db, redis, http, logging, telemetry
├─ services/                  # gateway ledger fraud sanctions insights notifier audit
│  └─ */chaos.py              # each service's pathologies, clearly marked
├─ tests/                     # black-box integration suite — a frozen contract
│  ├─ harness/                # load driver, pg_stat helpers, stream + container probes
│  └─ README.md               # methodology, markers, known-flaky list
├─ observability/             # dashboard generator (Foundation SDK); output ships via the Ansible role
├─ loadtest/                  # k6 scripts
└─ docs/
   ├─ roadmap/                       # the ten-phase learning plan, one file per phase
   │  ├─ intro.md                    # platform background, resource spine (read first)
   │  ├─ phase-0.md … phase-11.md    # one phase per file — status, message, concepts, build
   │  └─ summary.md                  # one-page status/sequencing overview of all phases
   ├─ phase-0a-test-suite-spec.md   # spec the test suite was written from
   ├─ phase-0-base-project-spec.md  # spec the base app was built from
   ├─ PATHOLOGIES.md                # ⚠️ spoilers — the answer key
   ├─ SLO.md                        # SLIs, SLOs, SLA, RTO/RPO, error budget policy
   ├─ CONCEPTS.md                   # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words
   ├─ INVESTIGATION.md              # the Phase 9 bottleneck hunt, written as an incident doc
   └─ PIPELINE-CHOICES.md           # Filebeat vs Logstash vs OpenTelemetry — when and why

---

Status

• Phase 0a — integration test suite, written first, red against nothing
• Phase 0b — base project implemented until the suite is green
• Phase 1 — deployment: Docker Compose + Traefik on a server
• Phase 2 — concepts, first trace end to end
• Phase 3 — distributed tracing across all seven services
• Phase 4 — structured logs, correlated with traces
• Phase 5 — metrics, RED, service map, exemplars
• Phase 6 — dashboards and alerts as code
• Phase 7 — k6 load testing, percentiles
• Phase 8 — SLIs, SLOs, error budgets, RTO/RPO
• Phase 9 — bottleneck hunt
• Phase 10 — Filebeat / Logstash / OpenTelemetry judgment
• Phase 11 — delivery

Full plan: docs/roadmap/ — start with intro.md, then the phase you're on, or summary.md for the one-page overview.

---

Non-goals

No authentication, no authorization, no input hardening, no unit tests, no Kubernetes, no production-grade storage. Ledgerline runs on a laptop under Docker Compose and is never exposed to a network you don't control.

The integration suite is deliberately the only test layer: it exists to pin down the system's observable behaviour — working and broken — not to verify the fake business logic.

Things deliberately left for a later iteration, and worth naming when discussing the project: tail-based sampling instead of 100%, Mimir instead of raw Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII redaction in the Collector, and multi-tenancy.