mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add project documentation and Claude instructions

Browse Source 
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.
This commit is contained in:
Movses Aslanyan
2026-09-05 08:11:00 +03:30
commit eb99cdbb8c
19 changed files with 3372 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+371
View File
@@ -0,0 +1,371 @@
# Ledgerline
A deliberately imperfect seven-service payments backend, built as a subject for
learning observability with OpenTelemetry and the Grafana stack.
> ### ⚠️ This system is broken on purpose
>
> Ledgerline ships with thirteen intentional performance and reliability
> pathologies: a missing index, a starved connection pool, an N+1 query, a
> blocking call inside an `async def`, a retry storm with no backoff, an
> unbounded cache, and more. They are toggled at runtime and documented in
> [`docs/PATHOLOGIES.md`](docs/PATHOLOGIES.md).
>
> **Do not fix them.** They are the entire point. If you are here to contribute,
> read [`CLAUDE.md`](CLAUDE.md) first.
The business logic is fake and does not need to be correct, secure, or
defensible. What it needs to be is *structurally* realistic: real network hops,
real SQL, real async message passing, real contention, and real latency
distributions — so that instrumenting it teaches something true.
---
## Quickstart
Requires Docker and Docker Compose. Nothing else.
```bash
git clone <this-repo> && cd ledgerline
cp .env.example .env
make up # start postgres, redis, and the seven services
make seed # schema + 250k transactions across 800 accounts (~60s)
make smoke # end-to-end sanity check
make test-contract # fast functional suite (~60s)
```
Then open **http://localhost:8000**.
The observability backend lives in a separate Ansible repo and is deployed to its
own host. From Phase 1 onward:
```bash
# in observability-ansible/
ansible-playbook playbooks/grafana-stack.yml
```
Then point Ledgerline at it via `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`.
Reset everything, including volumes:
```bash
make reset
```
---
## Architecture
```mermaid
flowchart LR
B[browser] --> GW[gateway :8000]
GW --> LD[ledger :8001]
GW --> FR[fraud :8002]
FR --> SA[sanctions :8003]
GW --> IN[insights :8004]
GW -- XADD --> ST[(redis stream)]
ST --> NO[notifier :8005]
ST --> AU[audit worker]
LD --> PG[(postgres)]
SA --> PG
IN --> PG
NO --> PG
AU --> PG
IN --> RD[(redis cache)]
```
| Service | Port | Role | Stores |
|---|---|---|---|
| `gateway` | 8000 | Public API, serves the frontend, orchestrates everything | Redis (flags) |
| `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
| `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
| `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow vendor" | Postgres (read) |
| `insights` | 8004 | 120-day analytics over transaction history | Postgres + Redis |
| `notifier` | 8005 | HTTP API **and** a Redis Stream consumer | Postgres |
| `audit` | — | Worker only; second consumer group on the same stream | Postgres |
Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz`
(dependency check).
**Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no
automatic trace-context propagation, so Phase 2 has to implement it by hand.
That's the most valuable exercise in the project. Don't swap it out.
### The primary request flow
`POST /api/transactions` produces a trace roughly four levels deep with one
async fork:
```
gateway
├─ ledger create transaction (pending, row lock on account)
├─ fraud score
│ └─ sanctions watchlist lookup
├─ ledger settle or reject
├─ redis XADD stream:transactions ──┬─ notifier (consumer group)
│ └─ audit (consumer group)
└─ insights cached account summary
```
The second flow, `GET /api/accounts/{id}/insights`, is a read-heavy aggregation
and behaves very differently under load.
---
## Chaos flags
Pathologies are controlled at runtime via a Redis hash (`chaos:flags`), read on
every request — **no restart needed**. Toggle them from the frontend's chaos
panel, the API, or the Makefile.
```bash
make chaos-clean # all off — the baseline
make chaos-light # a mild, realistic amount of pain
make chaos-nightmare # all thirteen on
make chaos-random # 3–6 flags, seed never logged
```
```bash
curl -s localhost:8000/api/admin/chaos | jq
curl -sX POST localhost:8000/api/admin/chaos \
-H 'content-type: application/json' \
-d '{"flag":"slow_sanctions","enabled":true}'
```
Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`,
`pool_starvation`, `row_lock_contention`, `blocking_event_loop`, `retry_storm`,
`no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`,
`chatty_health`.
> **During Phase 8, don't open the chaos panel and don't call
> `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the
> seed; the point is to find the pathologies from telemetry alone. The panel
> will happily tell you the answer, which is the one thing you don't want.
---
## Make targets
| Target | Effect |
|---|---|
| `up` / `down` | Start / stop the application stack |
| `logs` | `docker compose logs -f` |
| `seed` | Apply schema and seed 250k transactions |
| `reset` | `down -v` → `up` → `seed` |
| `smoke` | One transaction + one insights call, assert 200s |
| `test` | Full suite except the slow markers |
| `test-contract` | Functional + seed + guard assertions, < 60 s |
| `test-baseline` | Is the clean system genuinely healthy |
| `test-pathology` | One differential test per pathology — ⚠️ answer key, see below |
| `test-slow` | The soak-ish pathology tests |
| `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile |
| `fix-index` / `drop-index` | Add or remove the deliberately missing indexes |
| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 5+* |
| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 6+* |
---
## Testing
Phase 0 was built **test-first, by two separate agents**: one wrote `tests/`
from `docs/phase-0a-test-suite-spec.md` without seeing any implementation, the
second implemented the services until the suite went green without being allowed
to touch a test.
The suite is **black-box**. It imports nothing from the application and talks to
the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals,
not mechanics — there are no references to function names, module paths, or SQL
strings anywhere in it.
Two halves:
- **Working state.** A transaction really traverses all seven services (stop
`sanctions` and transactions must fail — that's the test that catches a
collapsed monolith); async work really happens off the request path via two
independent consumer groups; the seed data is really 250k rows, really skewed
toward five hot accounts, and really contains all five varieties of poison row.
- **Broken state.** One differential test per pathology, asserting the system
measurably degrades when a flag is on versus off. Assertions favour signals
that are hard to fake: `pg_stat_statements` query counts for the N+1,
`EXPLAIN` plan node types for the missing index, Redis consumer-group lag for
the slow consumer, container RSS for the memory leak, throughput plateaus for
pool starvation. A `sleep()` won't satisfy those.
```bash
make up && make seed # prerequisite — the suite runs against a live stack
make test-contract # fast
make test # everything except slow
```
> ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so
> its output is a complete answer key. Don't run it during the Phase 8
> bottleneck hunt.
`tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no
`prometheus_client`, no structured logging, no `/metrics`. **Delete it at the
start of Phase 2**; it has done its job by then.
Some pathology assertions currently count log lines, because plain text is the
only cross-service observability surface that exists at Phase 0. That is the
deficiency this project sets out to fix — those assertions get rewritten against
structured logs in Phase 3 and span data in Phase 4, and become both simpler and
more robust. See `tests/README.md`.
---
## Ports
| | Port | |
|---|---|---|
| Frontend / gateway | 8000 | |
| `ledger` … `notifier` | 8001–8005 | |
| Postgres | 5432 | `ledgerline` / `ledgerline` |
| Redis | 6379 | |
On the observability host (deployed by `observability-ansible`):
| | Port | |
|---|---|---|
| Grafana | 3000 | credentials from `group_vars/all.yml` |
| Prometheus | 9090 | remote-write receiver enabled |
| Alertmanager | 9093 | |
| Loki | 3100 | |
| Tempo | 3200 | |
| Alloy UI | 12345 | component graph — check here first when data is missing |
| OTLP (Alloy) | 4317 / 4318 | gRPC / HTTP |
| Node Exporter | 9100 | |
---
## Observability stack
The backend is **not** part of this repo. It is deployed by a separate Ansible
project, `observability-ansible`, onto a target host:
```
Ledgerline /metrics ──────scrape──────────────┐
container stdout ──docker.sock──▶ Alloy ──▶ Loki
Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo
│ span metrics + exemplars
▼ (remote_write)
Prometheus ◀────────┘
│
Alertmanager ──▶ Receivers
│
Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above
Correlation key: trace_id / exemplars
```
RED metrics and the service graph are generated by **Tempo's metrics-generator**,
not by collector connectors — do not add `spanmetrics`/`servicegraph` to Alloy.
Three constraints inherited from that platform, worth knowing before you debug
anything:
- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
accepted and silently dropped — by design; the platform's diagram labels that
input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
case-sensitively.** Log lines must carry lowercase `trace_id`. OTel's logging
auto-instrumentation emits `otelTraceID` and will not match.
- **The platform provisions datasources only.** Every dashboard here is generated
from code with the Grafana Foundation SDK and shipped through the Ansible role,
so `ansible-playbook` rebuilds them from scratch. **Never save a dashboard from
the Grafana UI** — it won't survive the next run.
Set `OTEL_EXPORTER_OTLP_ENDPOINT` (and friends) in `.env` to point at the Alloy
host. Nothing observability-related is hardcoded.
---
## The five-minute demo
Rehearse this before showing anyone.
1. Delete every dashboard in the Grafana UI, then re-run
`ansible-playbook playbooks/grafana-stack.yml` — platform and dashboards both
rebuilt from code, one command.
2. Submit one transaction from the frontend. Follow that single request through
Grafana: RED dashboard → latency spike → exemplar → trace → the slow span →
that span's logs → back to the trace. One request, three signals, no
searching.
3. `make chaos-nightmare && make load-stress` — watch p95 diverge from p50, the
service map turn red at a specific node, the error budget burn, and the
fast-burn alert fire.
4. Walk through [`docs/INVESTIGATION.md`](docs/INVESTIGATION.md) — how the
pathologies were found from telemetry alone.
---
## Repository layout
```
├─ CLAUDE.md # instructions for AI assistants — read before contributing
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile # one image, all Python services
├─ db/
│ ├─ schema.sql
│ ├─ optional_indexes.sql # the deliberately-omitted indexes
│ └─ seed_manifest.json # written by the seeder; the tests' source of truth
├─ scripts/ # seed.py, smoke.py
├─ common/ # settings, db, redis, http, logging, telemetry
├─ services/ # gateway ledger fraud sanctions insights notifier audit
│ └─ */chaos.py # each service's pathologies, clearly marked
├─ tests/ # black-box integration suite — a frozen contract
│ ├─ harness/ # load driver, pg_stat helpers, stream + container probes
│ └─ README.md # methodology, markers, known-flaky list
├─ observability/ # dashboard generator (Foundation SDK); output ships via the Ansible role
├─ loadtest/ # k6 scripts
└─ docs/
├─ roadmap/ # the ten-phase learning plan, one file per phase
│ ├─ intro.md # platform background, resource spine (read first)
│ ├─ phase-0.md … phase-10.md # one phase per file — status, message, concepts, build
│ └─ summary.md # one-page status/sequencing overview of all phases
├─ phase-0a-test-suite-spec.md # spec the test suite was written from
├─ phase-0-base-project-spec.md # spec the base app was built from
├─ PATHOLOGIES.md # ⚠️ spoilers — the answer key
├─ SLO.md # SLIs, SLOs, SLA, RTO/RPO, error budget policy
├─ CONCEPTS.md # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words
├─ INVESTIGATION.md # the Phase 8 bottleneck hunt, written as an incident doc
└─ PIPELINE-CHOICES.md # Filebeat vs Logstash vs OpenTelemetry — when and why
```
---
## Status
- [ ] **Phase 0a** — integration test suite, written first, red against nothing
- [ ] **Phase 0b** — base project implemented until the suite is green
- [ ] **Phase 1** — concepts, first trace end to end
- [ ] **Phase 2** — distributed tracing across all seven services
- [ ] **Phase 3** — structured logs, correlated with traces
- [ ] **Phase 4** — metrics, RED, service map, exemplars
- [ ] **Phase 5** — dashboards and alerts as code
- [ ] **Phase 6** — k6 load testing, percentiles
- [ ] **Phase 7** — SLIs, SLOs, error budgets, RTO/RPO
- [ ] **Phase 8** — bottleneck hunt
- [ ] **Phase 9** — Filebeat / Logstash / OpenTelemetry judgment
- [ ] **Phase 10** — delivery
Full plan: [`docs/roadmap/`](docs/roadmap/) — start with
[`intro.md`](docs/roadmap/intro.md), then the phase you're on, or
[`summary.md`](docs/roadmap/summary.md) for the one-page overview.
---
## Non-goals
No authentication, no authorization, no input hardening, no unit tests, no
Kubernetes, no production-grade storage. Ledgerline runs on a laptop under
Docker Compose and is never exposed to a network you don't control.
The integration suite is deliberately the *only* test layer: it exists to pin
down the system's observable behaviour — working and broken — not to verify the
fake business logic.
Things deliberately left for a later iteration, and worth naming when discussing
the project: tail-based sampling instead of 100%, Mimir instead of raw
Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII
redaction in the Collector, and multi-tenancy.