LedgerLine/README.md

# Ledgerline

A deliberately imperfect seven-service payments backend, built as a subject for
learning observability with OpenTelemetry and the Grafana stack.

> ### ⚠️ This system is broken on purpose
>
> Ledgerline ships with thirteen intentional performance and reliability
> pathologies: a missing index, a starved connection pool, an N+1 query, a
> blocking call inside an `async def`, a retry storm with no backoff, an
> unbounded cache, and more. They are toggled at runtime and documented in
> [`docs/PATHOLOGIES.md`](docs/PATHOLOGIES.md).
>
> **Do not fix them.** They are the entire point. If you are here to contribute,
> read [`CLAUDE.md`](CLAUDE.md) first.

The business logic is fake and does not need to be correct, secure, or
defensible. What it needs to be is *structurally* realistic: real network hops,
real SQL, real async message passing, real contention, and real latency
distributions — so that instrumenting it teaches something true.

---

## Quickstart

Requires Docker and Docker Compose. Nothing else.

```bash
git clone <this-repo> && cd ledgerline
cp .env.example .env
make up             # start postgres, redis, and the seven services
make seed           # schema + 250k transactions across 800 accounts (~60s)
make smoke          # end-to-end sanity check
make test-contract  # fast functional suite (~60s)
```

Then open **http://localhost:8000**.

The observability backend lives in a separate Ansible repo and is deployed to its
own host. From Phase 2 onward:

```bash
# in observability-ansible/
ansible-playbook playbooks/grafana-stack.yml
```

Then point Ledgerline at it via `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`.

Reset everything, including volumes:

```bash
make reset
```

---

## Architecture

```mermaid
flowchart LR
    B[browser] --> GW[gateway :8000]
    GW --> LD[ledger :8001]
    GW --> FR[fraud :8002]
    FR --> SA[sanctions :8003]
    GW --> IN[insights :8004]
    GW -- XADD --> ST[(redis stream)]
    ST --> NO[notifier :8005]
    ST --> AU[audit worker]
    LD --> PG[(postgres)]
    SA --> PG
    IN --> PG
    NO --> PG
    AU --> PG
    IN --> RD[(redis cache)]
```

| Service | Port | Role | Stores |
|---|---|---|---|
| `gateway` | 8000 | Public API, serves the frontend, orchestrates everything | Redis (flags) |
| `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
| `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
| `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow vendor" | Postgres (read) |
| `insights` | 8004 | 120-day analytics over transaction history | Postgres + Redis |
| `notifier` | 8005 | HTTP API **and** a Redis Stream consumer | Postgres |
| `audit` | — | Worker only; second consumer group on the same stream | Postgres |

Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz`
(dependency check).

**Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no
automatic trace-context propagation, so Phase 3 has to implement it by hand.
That's the most valuable exercise in the project. Don't swap it out.

### The primary request flow

`POST /api/transactions` produces a trace roughly four levels deep with one
async fork:

```
gateway
  ├─ ledger    create transaction (pending, row lock on account)
  ├─ fraud     score
  │    └─ sanctions   watchlist lookup
  ├─ ledger    settle or reject
  ├─ redis     XADD stream:transactions  ──┬─ notifier  (consumer group)
  │                                        └─ audit     (consumer group)
  └─ insights  cached account summary
```

The second flow, `GET /api/accounts/{id}/insights`, is a read-heavy aggregation
and behaves very differently under load.

---

## Chaos flags

Pathologies are controlled at runtime via a Redis hash (`chaos:flags`), read on
every request — **no restart needed**. Toggle them from the frontend's chaos
panel, the API, or the Makefile.

```bash
make chaos-clean       # all off — the baseline
make chaos-light       # a mild, realistic amount of pain
make chaos-nightmare   # all thirteen on
make chaos-random      # 3–6 flags, seed never logged
```

```bash
curl -s localhost:8000/api/admin/chaos | jq
curl -sX POST localhost:8000/api/admin/chaos \
  -H 'content-type: application/json' \
  -d '{"flag":"slow_sanctions","enabled":true}'
```

Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`,
`pool_starvation`, `row_lock_contention`, `blocking_event_loop`, `retry_storm`,
`no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`,
`chatty_health`.

> **During Phase 9, don't open the chaos panel and don't call
> `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the
> seed; the point is to find the pathologies from telemetry alone. The panel
> will happily tell you the answer, which is the one thing you don't want.

---

## Make targets

| Target | Effect |
|---|---|
| `up` / `down` | Start / stop the application stack |
| `logs` | `docker compose logs -f` |
| `seed` | Apply schema and seed 250k transactions |
| `reset` | `down -v` → `up` → `seed` |
| `smoke` | One transaction + one insights call, assert 200s |
| `test` | Full suite except the slow markers |
| `test-contract` | Functional + seed + guard assertions, < 60 s |
| `test-baseline` | Is the clean system genuinely healthy |
| `test-pathology` | One differential test per pathology — ⚠️ answer key, see below |
| `test-slow` | The soak-ish pathology tests |
| `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile |
| `fix-index` / `drop-index` | Add or remove the deliberately missing indexes |
| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 6+* |
| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 7+* |

---

## Testing

Phase 0 was built **test-first, by two separate agents**: one wrote `tests/`
from `docs/phase-0a-test-suite-spec.md` without seeing any implementation, the
second implemented the services until the suite went green without being allowed
to touch a test.

The suite is **black-box**. It imports nothing from the application and talks to
the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals,
not mechanics — there are no references to function names, module paths, or SQL
strings anywhere in it.

Two halves:

- **Working state.** A transaction really traverses all seven services (stop
  `sanctions` and transactions must fail — that's the test that catches a
  collapsed monolith); async work really happens off the request path via two
  independent consumer groups; the seed data is really 250k rows, really skewed
  toward five hot accounts, and really contains all five varieties of poison row.
- **Broken state.** One differential test per pathology, asserting the system
  measurably degrades when a flag is on versus off. Assertions favour signals
  that are hard to fake: `pg_stat_statements` query counts for the N+1,
  `EXPLAIN` plan node types for the missing index, Redis consumer-group lag for
  the slow consumer, container RSS for the memory leak, throughput plateaus for
  pool starvation. A `sleep()` won't satisfy those.

```bash
make up && make seed      # prerequisite — the suite runs against a live stack
make test-contract        # fast
make test                 # everything except slow
```

> ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so
> its output is a complete answer key. Don't run it during the Phase 9
> bottleneck hunt.

`tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no
`prometheus_client`, no structured logging, no `/metrics`. **Delete it at the
start of Phase 3**; it has done its job by then.

Some pathology assertions currently count log lines, because plain text is the
only cross-service observability surface that exists at Phase 0. That is the
deficiency this project sets out to fix — those assertions get rewritten against
structured logs in Phase 4 and span data in Phase 5, and become both simpler and
more robust. See `tests/README.md`.

---

## Ports

| | Port | |
|---|---|---|
| Frontend / gateway | 8000 | |
| `ledger` … `notifier` | 8001–8005 | |
| Postgres | 5432 | `ledgerline` / `ledgerline` |
| Redis | 6379 | |

On the observability host (deployed by `observability-ansible`):

| | Port | |
|---|---|---|
| Grafana | 3000 | credentials from `group_vars/all.yml` |
| Prometheus | 9090 | remote-write receiver enabled |
| Alertmanager | 9093 | |
| Loki | 3100 | |
| Tempo | 3200 | |
| Alloy UI | 12345 | component graph — check here first when data is missing |
| OTLP (Alloy) | 4317 / 4318 | gRPC / HTTP |
| Node Exporter | 9100 | |

---

## Observability stack

The backend is **not** part of this repo. It is deployed by a separate Ansible
project, `observability-ansible`, onto a target host:

```
Ledgerline /metrics ──────scrape──────────────┐
container stdout ──docker.sock──▶ Alloy ──▶ Loki
Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo
                                              │  span metrics + exemplars
                                              ▼         (remote_write)
                                         Prometheus ◀────────┘
                                              │
                                       Alertmanager ──▶ Receivers
                                              │
   Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above
          Correlation key: trace_id / exemplars
```

RED metrics and the service graph are generated by **Tempo's metrics-generator**,
not by collector connectors — do not add `spanmetrics`/`servicegraph` to Alloy.

Three constraints inherited from that platform, worth knowing before you debug
anything:

- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
  rejected with a 404 on `/v1/metrics` and `/v1/logs`, and the only sign is an
  export error in the sending app's logs. This is by design; the platform's diagram labels that
  input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
  each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
  case-sensitively.** Log lines must carry lowercase `trace_id`. OTel's logging
  auto-instrumentation emits `otelTraceID` and will not match.
- **The platform provisions datasources only.** Every dashboard here is generated
  from code with the Grafana Foundation SDK and shipped through the Ansible role,
  so `ansible-playbook` rebuilds them from scratch. **Never save a dashboard from
  the Grafana UI** — it won't survive the next run.

Set `OTEL_EXPORTER_OTLP_ENDPOINT` (and friends) in `.env` to point at the Alloy
host. Nothing observability-related is hardcoded.

---

## The five-minute demo

Rehearse this before showing anyone.

1. Delete every dashboard in the Grafana UI, then re-run
   `ansible-playbook playbooks/grafana-stack.yml` — platform and dashboards both
   rebuilt from code, one command.
2. Submit one transaction from the frontend. Follow that single request through
   Grafana: RED dashboard → latency spike → exemplar → trace → the slow span →
   that span's logs → back to the trace. One request, three signals, no
   searching.
3. `make chaos-nightmare && make load-stress` — watch p95 diverge from p50, the
   service map turn red at a specific node, the error budget burn, and the
   fast-burn alert fire.
4. Walk through [`docs/INVESTIGATION.md`](docs/INVESTIGATION.md) — how the
   pathologies were found from telemetry alone.

---

## Repository layout

```
├─ CLAUDE.md                  # instructions for AI assistants — read before contributing
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile                 # one image, all Python services
├─ db/
│  ├─ schema.sql
│  ├─ optional_indexes.sql    # the deliberately-omitted indexes
│  └─ seed_manifest.json      # written by the seeder; the tests' source of truth
├─ scripts/                   # seed.py, smoke.py
├─ common/                    # settings, db, redis, http, logging, telemetry
├─ services/                  # gateway ledger fraud sanctions insights notifier audit
│  └─ */chaos.py              # each service's pathologies, clearly marked
├─ tests/                     # black-box integration suite — a frozen contract
│  ├─ harness/                # load driver, pg_stat helpers, stream + container probes
│  └─ README.md               # methodology, markers, known-flaky list
├─ observability/             # dashboard generator (Foundation SDK); output ships via the Ansible role
├─ loadtest/                  # k6 scripts
└─ docs/
   ├─ roadmap/                       # the ten-phase learning plan, one file per phase
   │  ├─ intro.md                    # platform background, resource spine (read first)
   │  ├─ phase-0.md … phase-11.md    # one phase per file — status, message, concepts, build
   │  └─ summary.md                  # one-page status/sequencing overview of all phases
   ├─ phase-0a-test-suite-spec.md   # spec the test suite was written from
   ├─ phase-0-base-project-spec.md  # spec the base app was built from
   ├─ PATHOLOGIES.md                # ⚠️ spoilers — the answer key
   ├─ SLO.md                        # SLIs, SLOs, SLA, RTO/RPO, error budget policy
   ├─ CONCEPTS.md                   # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words
   ├─ INVESTIGATION.md              # the Phase 9 bottleneck hunt, written as an incident doc
   └─ PIPELINE-CHOICES.md           # Filebeat vs Logstash vs OpenTelemetry — when and why
```

---

## Status

- [ ] **Phase 0a** — integration test suite, written first, red against nothing
- [ ] **Phase 0b** — base project implemented until the suite is green
- [x] **Phase 1** — deployment: Docker Compose + Traefik on a server
- [ ] **Phase 2** — concepts, first trace end to end
- [ ] **Phase 3** — distributed tracing across all seven services
- [ ] **Phase 4** — structured logs, correlated with traces
- [ ] **Phase 5** — metrics, RED, service map, exemplars
- [ ] **Phase 6** — dashboards and alerts as code
- [ ] **Phase 7** — k6 load testing, percentiles
- [ ] **Phase 8** — SLIs, SLOs, error budgets, RTO/RPO
- [ ] **Phase 9** — bottleneck hunt
- [ ] **Phase 10** — Filebeat / Logstash / OpenTelemetry judgment
- [ ] **Phase 11** — delivery

Full plan: [`docs/roadmap/`](docs/roadmap/) — start with
[`intro.md`](docs/roadmap/intro.md), then the phase you're on, or
[`summary.md`](docs/roadmap/summary.md) for the one-page overview.

---

## Non-goals

No authentication, no authorization, no input hardening, no unit tests, no
Kubernetes, no production-grade storage. Ledgerline runs on a laptop under
Docker Compose and is never exposed to a network you don't control.

The integration suite is deliberately the *only* test layer: it exists to pin
down the system's observable behaviour — working and broken — not to verify the
fake business logic.

Things deliberately left for a later iteration, and worth naming when discussing
the project: tail-based sampling instead of 100%, Mimir instead of raw
Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII
redaction in the Collector, and multi-tenancy.