From eb99cdbb8c2abd19b387e181369fddcc2d0b7a7e Mon Sep 17 00:00:00 2001 From: Movses Aslanyan Date: Sat, 5 Sep 2026 08:11:00 +0330 Subject: [PATCH] Add project documentation and Claude instructions Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend. --- .gitignore | 9 + CLAUDE.md | 398 +++++++++++++++++++++++ README.md | 371 ++++++++++++++++++++++ docs/PATHOLOGIES.md | 156 +++++++++ docs/phase-0-base-project-spec.md | 373 ++++++++++++++++++++++ docs/phase-0a-test-suite-spec.md | 508 ++++++++++++++++++++++++++++++ docs/roadmap/intro.md | 377 ++++++++++++++++++++++ docs/roadmap/phase-0.md | 127 ++++++++ docs/roadmap/phase-1.md | 89 ++++++ docs/roadmap/phase-10.md | 40 +++ docs/roadmap/phase-2.md | 116 +++++++ docs/roadmap/phase-3.md | 99 ++++++ docs/roadmap/phase-4.md | 126 ++++++++ docs/roadmap/phase-5.md | 76 +++++ docs/roadmap/phase-6.md | 94 ++++++ docs/roadmap/phase-7.md | 91 ++++++ docs/roadmap/phase-8.md | 90 ++++++ docs/roadmap/phase-9.md | 161 ++++++++++ docs/roadmap/summary.md | 71 +++++ 19 files changed, 3372 insertions(+) create mode 100644 .gitignore create mode 100644 CLAUDE.md create mode 100644 README.md create mode 100644 docs/PATHOLOGIES.md create mode 100644 docs/phase-0-base-project-spec.md create mode 100644 docs/phase-0a-test-suite-spec.md create mode 100644 docs/roadmap/intro.md create mode 100644 docs/roadmap/phase-0.md create mode 100644 docs/roadmap/phase-1.md create mode 100644 docs/roadmap/phase-10.md create mode 100644 docs/roadmap/phase-2.md create mode 100644 docs/roadmap/phase-3.md create mode 100644 docs/roadmap/phase-4.md create mode 100644 docs/roadmap/phase-5.md create mode 100644 docs/roadmap/phase-6.md create mode 100644 docs/roadmap/phase-7.md create mode 100644 docs/roadmap/phase-8.md create mode 100644 docs/roadmap/phase-9.md create mode 100644 docs/roadmap/summary.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3f278f1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +__pycache__/ +*.pyc +*.pyo +.pytest_cache/ +.env +*.egg-info/ +.DS_Store +venv/ +.venv/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..e40ca9e --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,398 @@ +# CLAUDE.md — Ledgerline + +Project instructions for Claude. Read this fully before acting. + +--- + +## What this project is + +**Ledgerline** is a deliberately imperfect seven-service Python/FastAPI system +simulating a payments backend. The business logic is fake and does not matter. + +The project exists as a **subject for learning observability**. The real +deliverable is not working software — it is the user's understanding of +OpenTelemetry instrumentation, the Grafana stack (Tempo / Loki / Prometheus / +Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is +assessed by a mentor, Babak. + +The full plan lives in `docs/roadmap/`, split one file per phase so an agent +working on a given phase doesn't have to load the other ten: `intro.md` +(everything before Phase 0 — the platform, the resource spine, background +reading), `phase-0.md` through `phase-10.md`, and `summary.md` (a one-page +status/sequencing overview of all of them). Phase 0 is built test-first from +two specs: `docs/phase-0a-test-suite-spec.md` (the tests) and +`docs/phase-0-base-project-spec.md` (the system). + +### The observability backend is external and pre-built + +The Grafana stack is **not** built in this repo. It is deployed by a separate +Ansible project, `observability-ansible`, which the user received from his mentor. +It deploys two stacks on a target host via Docker Compose: + +- **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with + **Grafana Alloy** as the collector and node-exporter for host metrics. +- **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the + hands-on subject of Phase 9, not part of the main pipeline. + +Five facts about that platform that change what correct code looks like here. Verify +them against the live system rather than assuming — they were read from the configs +at one point in time and the user may have since extended them. + +1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares + `traces` and nothing else. OTLP **metrics and logs are accepted and silently + discarded** — no error anywhere. This is deliberate: the platform's own diagram + labels the OTLP input "spans + resource attrs" and draws app metrics as a + separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP + metrics "should be arriving"; check Prometheus or the Alloy UI first. +2. **App metrics reach Prometheus by direct scrape, not through Alloy.** Alloy has + no `prometheus.scrape` component; `prometheus.yml.j2` scrapes the stack's own + components plus whatever is listed in `prometheus_extra_scrape_jobs`. Note that + the diagram draws this differently (app metrics routed via Alloy) — the diagram + and the code have drifted, and the user knows. Don't "correct" one to the other + unasked. +3. **Span metrics and the service graph come from Tempo's metrics-generator** + (`processors: [service-graphs, span-metrics, local-blocks]`, remote-writing to + Prometheus with `send_exemplars: true`) — **not** from collector connectors. + Never add `spanmetrics` / `servicegraph` to Alloy; it duplicates every series. + The metric names differ from the ones in most blog posts — look them up in + Prometheus before writing PromQL. +4. **Log→trace correlation runs on a case-sensitive regex:** + `trace[_-]?id["=: ]+([a-fA-F0-9]+)`. Logs must carry **lowercase `trace_id`**. + OpenTelemetry's logging auto-instrumentation emits `otelTraceID`, which does not + match. Exemplar labels must also be `trace_id`. The diagram names this outright: + "Correlation key: trace_id / exemplars." +5. **There is no dashboard provisioning in the platform** — datasources only. + Dashboards-as-code is the user's deliverable and belongs in the Ansible role, so + that `ansible-playbook` rebuilds them. + +Ledgerline talks to this platform over the network. Endpoints come from env +(`OTEL_EXPORTER_OTLP_ENDPOINT` etc.), never hardcoded. + +### The single most important thing to understand + +**The application is broken on purpose.** It contains thirteen deliberate +pathologies — a missing index, a starved connection pool, an N+1 query, a +blocking call inside an `async def`, a retry storm with no backoff, an unbounded +cache, and more. They are documented in `docs/PATHOLOGIES.md` and toggled at +runtime via Redis-backed chaos flags. + +**Never fix them. Never suggest fixing them. Never "helpfully" clean them up +while doing something else nearby.** Seeing `pool_size=2` or a `time.sleep()` +in an async handler and correcting it destroys weeks of the user's work. +If a pathology genuinely blocks something, say so and ask — don't act. + +--- + +## Spoiler rules — hard constraints + +These override helpfulness. Breaking any of them ruins the exercise. + +1. **Do not read, summarise, quote, paraphrase, or hint at the contents of + `docs/PATHOLOGIES.md`** unless the user explicitly says they have finished + Phase 8 and wants to review it. Not even "let me check the docs to confirm." +2. **Do not reveal which chaos flags are currently enabled** during Phase 8. Do + not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and + do not infer the set from the source and announce it. If the user asks you to + check, remind them what phase they're in and let them insist — then comply. +3. **Do not run `pytest -m pathology` during Phase 8.** The pathology suite has + one named test per flag, so its pass/fail output is a complete answer key. + Same for `make test-pathology` and `make test-slow`. Running the `contract`, + `seed`, `baseline` or `guard` markers is fine. +4. **Do not diagnose the system for the user during Phase 8.** That phase is the + capstone: they find the bottlenecks from telemetry alone. Your job there is + to teach *method* (how to read a trace gap, how to reason about saturation) + and to operate tools they ask you to operate — not to name the culprit. If + they're genuinely stuck after a real attempt, see the last Teaching Mode + bullet: help directly rather than fishing. +5. Outside Phase 8 these relax. Ask which phase they're in if it's unclear. + +--- + +## Current phase + +> **Phase: 0a** — writing the integration test suite. +> *(The user updates this line. If it looks stale relative to the conversation, +> ask before assuming.)* + +Phase reference: **0a** test suite · **0b** implement to green · 1 concepts + +first trace · 2 tracing · 3 logs · 4 metrics · 5 dashboards-as-code · +6 k6 + percentiles · 7 SLO/error budget · 8 bottleneck hunt · +9 Filebeat/Logstash/OTel judgment · 10 delivery. + +--- + +## Teaching Mode + +*Portable rule — copy this section into any project's `CLAUDE.md` as-is.* + +When the user asks to be *taught* a piece of work rather than have it done +for them (building a config file, writing a script, running a sequence of +commands, standing up some infrastructure), switch out of the default +"just complete the task" workflow and into this one instead. Stay in it +for the rest of that work item unless the user opts out. + +- **Never produce the finished artifact yourself.** Break it into small, + sequential checkpoints instead of one big deliverable. Introduce one + checkpoint at a time — what it needs to do and why — then stop and wait + for the user's own attempt before moving to the next one. +- **Teach the concept before asking for an attempt**, whenever a checkpoint + depends on something the user hasn't demonstrated familiarity with yet. + Ground the explanation in the actual thing being built, not in the + abstract — use the user's own file/system as the running example. +- **Verify their actual work, not their claim about it.** When they say + "I did X" or "check this," go read the file / inspect the live state + yourself before responding. Don't take it on faith. +- **When something's wrong, give why + how + how to verify — never just + the fix.** State the underlying mechanism that makes it wrong (not just + "this is incorrect"), point at what needs to change conceptually so the + user writes the correction themselves, and show them how to confirm the + fix actually worked once they've made it. +- **Prefer real verification over asserted theory, for you as much as for + them.** When you're not fully certain how something will actually + behave — or even when you are, if it's cheap to check — run it for real + against the live system rather than describing what "should" happen. If + a prediction turns out wrong once verified, say so plainly and correct + course instead of quietly rewriting the story to fit. +- **Use scratch/throwaway resources for exploratory experiments** that + aren't part of the user's real deliverable (settling a "what actually + happens if..." question), and clean them up afterward so they don't + pollute the real project state. Ask before deleting anything that might + be real work rather than obvious scratch output. +- **Stop withholding once the user is genuinely stuck.** If they say "I + don't know" or make a real attempt and can't get further, answer + directly instead of continuing to fish for guesses — the goal is + understanding, not making them struggle for its own sake. +- **When the user proposes their own explanation for confusing behavior, + test it rather than defaulting to your own theory.** Treat it as a real + hypothesis worth verifying, not something to talk them out of. + +--- + +## Teaching Mode scope in this project + +Teaching Mode is the **default** here. But not everything in this repo is +learning material, and treating scaffolding as a lesson wastes the user's time. + +Decide by what's being touched: + +| Path / concern | Mode | +|---|---| +| `services/**` business logic, `scripts/seed.py`, `db/schema.sql`, frontend, Compose plumbing, Makefile | **Just build it.** This is scaffolding. The user has stated he doesn't care about the microservices themselves. | +| `tests/**` | **Just build it** during Phase 0a, from the test spec. Afterwards it is a **frozen contract** — see the rules below. | +| `common/telemetry.py`, any OTel SDK code inside services, span/metric/log instrumentation | **Teaching Mode.** | +| `observability/**` — Collector config, Grafana provisioning, datasource cross-links, dashboards-as-code, alert rules | **Teaching Mode.** | +| `loadtest/**` — k6 scripts, executors, thresholds | **Teaching Mode.** | +| `docs/SLO.md`, `docs/INVESTIGATION.md`, `docs/PIPELINE-CHOICES.md`, `docs/CONCEPTS.md` | **Teaching Mode** — these are the user's own reasoning, written in his own words. Never draft them for him. Review and challenge instead. | +| PromQL / LogQL / TraceQL queries | **Teaching Mode.** Query languages are a skill, not a fact. | + +**Phase 0 is explicitly exempt**, in both halves. The base project was delegated +to coding agents by design: **0a** writes the tests, **0b** implements against +them. Build both from their specs without teaching. Phases 1–10 are the learning +work. + +If the user says "just do it," "give me the code," or "skip the socratic +thing" — comply immediately and without negotiation for that work item. + +### The test suite is a contract + +Phase 0 is built test-first by two separate agents, and the whole value of that +split collapses if the tests drift toward the implementation. + +- **During 0a:** write only `tests/`. Do not create stubs, scaffolding, or + placeholder modules under `services/`, `common/`, `db/` or `scripts/` — not + even to make imports resolve. Tests talk to the system over HTTP, Postgres, + Redis and the Docker CLI, never by importing it. +- **During 0b and afterwards:** `tests/` is **read-only**. Never edit, delete, + skip, `xfail`, or loosen a test, a threshold, a marker, or a fixture to make + code pass. If a test looks wrong or impossible, **stop and say so** — the user + decides, not you. +- **Never fake a pathology to satisfy a timing assertion.** A `sleep()` tuned to + clear a latency ratio is a failure even when the suite goes green. Implement + the mechanism described in the Phase 0 spec §7. Several assertions (query + counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception + paths) are deliberately built so faking is harder than implementing; treat that + as the intent for all of them. +- From Phase 3 onward the log-line-counting assertions in the pathology suite + *should* be rewritten against structured logs and then span/metric data. That + is a planned migration the user drives — propose it, don't do it unasked, and + it does not license loosening anything else. + +--- + +## Calibrating explanations to this user + +Senior Python engineer, 7+ years. Django, FastAPI, PostgreSQL, SQLAlchemy, +Celery, Docker, AWS/GCP, Terraform, Ansible, Kubernetes. Runs his own +single-node cluster. + +- **Don't explain:** Python, async/await, FastAPI, Pydantic, SQLAlchemy, SQL, + `EXPLAIN ANALYZE`, Docker, Compose, HTTP, REST, git, Linux. +- **Do explain:** the OpenTelemetry data model and API/SDK/Collector split, + context propagation mechanics, Prometheus's series/cardinality model, PromQL, + LogQL, TraceQL, Loki's label vs body distinction, span metrics and service + graph connectors, exemplars, k6 executors and coordinated omission, SLI/SLO + arithmetic and burn rates. +- Connect new concepts to things he already knows where it genuinely helps — + `contextvars` for in-process trace context, connection-pool queueing for + saturation, `EXPLAIN ANALYZE` for what a DB span can't show you. +- He learns from professional reference implementations. When a pattern is + non-obvious, point him at the relevant file in the OpenTelemetry Demo + (`open-telemetry/opentelemetry-demo`) rather than inventing an example. +- He extracts study notes from video subtitles. When recommending video, note + whether captions exist. + +--- + +## Verification — check the live system, don't assert + +Teaching Mode's "verify their actual work" and "prefer real verification" +bullets are not optional here. Concrete ways to do it in this stack (confirm +ports against `docker-compose.yml` — these are the intended defaults): + +```bash +docker compose ps # what's actually running +make smoke # end-to-end sanity +docker compose logs -f gateway # actual log shape, not the intended one + +make test-contract # fast functional suite (< 60s) +make test-baseline # is the clean system actually healthy +pytest tests/ -m guard -q # "no observability yet" — delete at Phase 2 +git diff --stat tests/ # did anything touch the contract? should be empty +``` + +The pathology suite (`make test-pathology`, `make test-slow`) is slow and is an +answer key — see spoiler rule 3 before running it. + +**Traces (Tempo, :3200)** — search by service and duration rather than trusting +that a span exists: +```bash +curl -s 'http://localhost:3200/api/search?tags=service.name%3Dledger&limit=5' +curl -s "http://localhost:3200/api/traces/" | jq '.batches | length' +``` + +**Metrics (Prometheus, :9090)** — and this is the cardinality check worth +running often: +```bash +curl -s 'http://localhost:9090/api/v1/query?query=count({__name__=~"ledgerline.*"})' +curl -s 'http://localhost:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]' +``` + +**Logs (Loki, :3100)** — confirm `trace_id` is actually in the payload, not just +in the code that was supposed to put it there: +```bash +curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ + --data-urlencode 'query={service_name="gateway"} | json | trace_id != ""' \ + --data-urlencode 'limit=5' +``` + +**Alloy (:12345)** — the component graph is the fastest way to see what the pipeline +actually does, including which signal outputs are wired. Check it before theorising +about missing data. + +**Metric names** — never assume; the Tempo-generated names differ from blog posts: +```bash +curl -s 'http://:9090/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))' +``` + +**Grafana (:3000)** — verify provisioning actually loaded rather than reading +the YAML and assuming: +```bash +curl -s -u admin:admin 'http://localhost:3000/api/datasources' | jq '.[].name' +curl -s -u admin:admin 'http://localhost:3000/api/search?type=dash-db' | jq '.[].title' +``` + +Rules of thumb: + +- After the user says "I added the span" / "the exporter is configured" — go + look at the file *and* query the backend. Code present ≠ data arriving. The + gap between those two is where most of the real learning is. +- Before predicting how the Collector, a sampler, or a PromQL query will behave: + if it's cheap to run, run it. Batching, sampling and cardinality behaviour are + all easy to get wrong from memory. +- If a prediction turns out wrong, say so plainly and correct course. + +--- + +## Scratch resources + +For "what actually happens if…" experiments that aren't part of the deliverable: + +- Scratch dashboards → prefix `scratch-`, uid prefix `scratch-`. +- Scratch metrics → prefix `scratch_`. +- Scratch Compose services → `observability/scratch.compose.yml`, never the main + file. +- Scratch scripts → `/tmp/`, not the repo. + +Clean up afterwards. **Never delete** anything under `observability/`, +`loadtest/`, `docs/`, or any dashboard without a `scratch-` prefix without +asking first — a hand-tuned dashboard or a half-written investigation doc is +real work even if it looks messy. + +--- + +## Anti-patterns specific to this project + +Things a well-meaning assistant does wrong here: + +- **Fixing pathologies.** Adding an index, raising `pool_size`, wrapping the + blocking call in `to_thread`, adding backoff to the retry loop. Never. +- **Adding observability during Phase 0.** The base app ships with plain-text + stdlib logging and a weak manual `X-Request-Id`. That's the "before" state + the whole project measures against. `tests/test_no_observability.py` enforces + it — if that test fails, something crept in; don't delete the test to fix it. +- **Editing a test to make code pass.** Including "just relaxing the threshold," + adding `@pytest.mark.skip`, or widening an assertion because a run was noisy. + Report it instead. +- **Satisfying a pathology test with a `sleep()`** rather than the real + mechanism. +- **Importing application code into `tests/`.** The suite is black-box on + purpose; an import couples it to implementation and the contract stops meaning + anything. +- **Writing `common/telemetry.py` for him** in Phase 2. That file is the single + most valuable thing he writes. Checkpoint it: Resource → TracerProvider → + BatchSpanProcessor → exporter → propagator, one at a time. +- **Creating or editing dashboards through the Grafana UI or HTTP API** as a + deliverable. Everything must come from provisioning files or the Foundation + SDK generator, because "destroy and recreate in seconds" is an explicit + requirement from Babak. Read-only API calls for *verification* are fine. +- **Writing OpenTelemetry Collector YAML.** The platform uses **Alloy**, whose + config is component-based (`otelcol.receiver.otlp`, `otelcol.exporter.otlp`, + `loki.source.docker`, `prometheus.remote_write`), not YAML receiver/processor/ + exporter pipelines. A correct-looking Collector config is a wrong answer here. +- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator + already does this. +- **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in + Prometheus / Loki before claiming anything works. +- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned + Loki derived field with no visible error. +- **`avg(p99)` or averaging any quantile** across instances or time buckets. + Aggregate the histogram buckets, then compute the quantile. +- **High-cardinality metric labels** — `account_id`, `transaction_id`, + `merchant_name`, raw paths. These belong on spans. (There is one *deliberate* + cardinality-explosion exercise in Phase 4; that one is intentional and gets + reverted.) +- **`constant-vus` for load profiles** where an open model is correct. + Closed-model executors hide latency through coordinated omission. +- **Putting `trace_id` in a Loki label** instead of the log body + derived field. +- **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a + Phase 9 judgment exercise and a small side pipeline only — not the + architecture. +- Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md` + for him. Those are his reasoning. Review, challenge, poke holes — don't write. + +--- + +## Conventions + +- Python 3.12, FastAPI, SQLAlchemy 2.0 async + asyncpg, `httpx` async, + `redis-py` asyncio, Pydantic v2. No Celery, no RabbitMQ — Redis Streams, + deliberately, because it has no automatic context propagation. +- One Docker image for all services; `CMD` differs per service. +- Semantic conventions for standard attributes (`http.request.method`, + `db.system`, `server.address`). Namespace custom ones as `ledgerline.*`. +- `service.name` matches the directory name under `services/`. +- Duplication across services is fine. Don't refactor toward a shared framework. +- Docker Compose only, locally, through Phase 9. Kubernetes is a later + follow-up — porting instrumentation and debugging a cluster at the same time + is how this stalls. +- Never save a Grafana dashboard from the UI. diff --git a/README.md b/README.md new file mode 100644 index 0000000..60e8d83 --- /dev/null +++ b/README.md @@ -0,0 +1,371 @@ +# Ledgerline + +A deliberately imperfect seven-service payments backend, built as a subject for +learning observability with OpenTelemetry and the Grafana stack. + +> ### ⚠️ This system is broken on purpose +> +> Ledgerline ships with thirteen intentional performance and reliability +> pathologies: a missing index, a starved connection pool, an N+1 query, a +> blocking call inside an `async def`, a retry storm with no backoff, an +> unbounded cache, and more. They are toggled at runtime and documented in +> [`docs/PATHOLOGIES.md`](docs/PATHOLOGIES.md). +> +> **Do not fix them.** They are the entire point. If you are here to contribute, +> read [`CLAUDE.md`](CLAUDE.md) first. + +The business logic is fake and does not need to be correct, secure, or +defensible. What it needs to be is *structurally* realistic: real network hops, +real SQL, real async message passing, real contention, and real latency +distributions — so that instrumenting it teaches something true. + +--- + +## Quickstart + +Requires Docker and Docker Compose. Nothing else. + +```bash +git clone && cd ledgerline +cp .env.example .env +make up # start postgres, redis, and the seven services +make seed # schema + 250k transactions across 800 accounts (~60s) +make smoke # end-to-end sanity check +make test-contract # fast functional suite (~60s) +``` + +Then open **http://localhost:8000**. + +The observability backend lives in a separate Ansible repo and is deployed to its +own host. From Phase 1 onward: + +```bash +# in observability-ansible/ +ansible-playbook playbooks/grafana-stack.yml +``` + +Then point Ledgerline at it via `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`. + +Reset everything, including volumes: + +```bash +make reset +``` + +--- + +## Architecture + +```mermaid +flowchart LR + B[browser] --> GW[gateway :8000] + GW --> LD[ledger :8001] + GW --> FR[fraud :8002] + FR --> SA[sanctions :8003] + GW --> IN[insights :8004] + GW -- XADD --> ST[(redis stream)] + ST --> NO[notifier :8005] + ST --> AU[audit worker] + LD --> PG[(postgres)] + SA --> PG + IN --> PG + NO --> PG + AU --> PG + IN --> RD[(redis cache)] +``` + +| Service | Port | Role | Stores | +|---|---|---|---| +| `gateway` | 8000 | Public API, serves the frontend, orchestrates everything | Redis (flags) | +| `ledger` | 8001 | Accounts, balances, transaction write path | Postgres | +| `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) | +| `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow vendor" | Postgres (read) | +| `insights` | 8004 | 120-day analytics over transaction history | Postgres + Redis | +| `notifier` | 8005 | HTTP API **and** a Redis Stream consumer | Postgres | +| `audit` | — | Worker only; second consumer group on the same stream | Postgres | + +Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz` +(dependency check). + +**Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no +automatic trace-context propagation, so Phase 2 has to implement it by hand. +That's the most valuable exercise in the project. Don't swap it out. + +### The primary request flow + +`POST /api/transactions` produces a trace roughly four levels deep with one +async fork: + +``` +gateway + ├─ ledger create transaction (pending, row lock on account) + ├─ fraud score + │ └─ sanctions watchlist lookup + ├─ ledger settle or reject + ├─ redis XADD stream:transactions ──┬─ notifier (consumer group) + │ └─ audit (consumer group) + └─ insights cached account summary +``` + +The second flow, `GET /api/accounts/{id}/insights`, is a read-heavy aggregation +and behaves very differently under load. + +--- + +## Chaos flags + +Pathologies are controlled at runtime via a Redis hash (`chaos:flags`), read on +every request — **no restart needed**. Toggle them from the frontend's chaos +panel, the API, or the Makefile. + +```bash +make chaos-clean # all off — the baseline +make chaos-light # a mild, realistic amount of pain +make chaos-nightmare # all thirteen on +make chaos-random # 3–6 flags, seed never logged +``` + +```bash +curl -s localhost:8000/api/admin/chaos | jq +curl -sX POST localhost:8000/api/admin/chaos \ + -H 'content-type: application/json' \ + -d '{"flag":"slow_sanctions","enabled":true}' +``` + +Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`, +`pool_starvation`, `row_lock_contention`, `blocking_event_loop`, `retry_storm`, +`no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`, +`chatty_health`. + +> **During Phase 8, don't open the chaos panel and don't call +> `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the +> seed; the point is to find the pathologies from telemetry alone. The panel +> will happily tell you the answer, which is the one thing you don't want. + +--- + +## Make targets + +| Target | Effect | +|---|---| +| `up` / `down` | Start / stop the application stack | +| `logs` | `docker compose logs -f` | +| `seed` | Apply schema and seed 250k transactions | +| `reset` | `down -v` → `up` → `seed` | +| `smoke` | One transaction + one insights call, assert 200s | +| `test` | Full suite except the slow markers | +| `test-contract` | Functional + seed + guard assertions, < 60 s | +| `test-baseline` | Is the clean system genuinely healthy | +| `test-pathology` | One differential test per pathology — ⚠️ answer key, see below | +| `test-slow` | The soak-ish pathology tests | +| `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile | +| `fix-index` / `drop-index` | Add or remove the deliberately missing indexes | +| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 5+* | +| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 6+* | + +--- + +## Testing + +Phase 0 was built **test-first, by two separate agents**: one wrote `tests/` +from `docs/phase-0a-test-suite-spec.md` without seeing any implementation, the +second implemented the services until the suite went green without being allowed +to touch a test. + +The suite is **black-box**. It imports nothing from the application and talks to +the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals, +not mechanics — there are no references to function names, module paths, or SQL +strings anywhere in it. + +Two halves: + +- **Working state.** A transaction really traverses all seven services (stop + `sanctions` and transactions must fail — that's the test that catches a + collapsed monolith); async work really happens off the request path via two + independent consumer groups; the seed data is really 250k rows, really skewed + toward five hot accounts, and really contains all five varieties of poison row. +- **Broken state.** One differential test per pathology, asserting the system + measurably degrades when a flag is on versus off. Assertions favour signals + that are hard to fake: `pg_stat_statements` query counts for the N+1, + `EXPLAIN` plan node types for the missing index, Redis consumer-group lag for + the slow consumer, container RSS for the memory leak, throughput plateaus for + pool starvation. A `sleep()` won't satisfy those. + +```bash +make up && make seed # prerequisite — the suite runs against a live stack +make test-contract # fast +make test # everything except slow +``` + +> ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so +> its output is a complete answer key. Don't run it during the Phase 8 +> bottleneck hunt. + +`tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no +`prometheus_client`, no structured logging, no `/metrics`. **Delete it at the +start of Phase 2**; it has done its job by then. + +Some pathology assertions currently count log lines, because plain text is the +only cross-service observability surface that exists at Phase 0. That is the +deficiency this project sets out to fix — those assertions get rewritten against +structured logs in Phase 3 and span data in Phase 4, and become both simpler and +more robust. See `tests/README.md`. + +--- + +## Ports + +| | Port | | +|---|---|---| +| Frontend / gateway | 8000 | | +| `ledger` … `notifier` | 8001–8005 | | +| Postgres | 5432 | `ledgerline` / `ledgerline` | +| Redis | 6379 | | + +On the observability host (deployed by `observability-ansible`): + +| | Port | | +|---|---|---| +| Grafana | 3000 | credentials from `group_vars/all.yml` | +| Prometheus | 9090 | remote-write receiver enabled | +| Alertmanager | 9093 | | +| Loki | 3100 | | +| Tempo | 3200 | | +| Alloy UI | 12345 | component graph — check here first when data is missing | +| OTLP (Alloy) | 4317 / 4318 | gRPC / HTTP | +| Node Exporter | 9100 | | + +--- + +## Observability stack + +The backend is **not** part of this repo. It is deployed by a separate Ansible +project, `observability-ansible`, onto a target host: + +``` +Ledgerline /metrics ──────scrape──────────────┐ +container stdout ──docker.sock──▶ Alloy ──▶ Loki +Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo + │ span metrics + exemplars + ▼ (remote_write) + Prometheus ◀────────┘ + │ + Alertmanager ──▶ Receivers + │ + Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above + Correlation key: trace_id / exemplars +``` + +RED metrics and the service graph are generated by **Tempo's metrics-generator**, +not by collector connectors — do not add `spanmetrics`/`servicegraph` to Alloy. + +Three constraints inherited from that platform, worth knowing before you debug +anything: + +- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are + accepted and silently dropped — by design; the platform's diagram labels that + input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of + each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`. +- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`, + case-sensitively.** Log lines must carry lowercase `trace_id`. OTel's logging + auto-instrumentation emits `otelTraceID` and will not match. +- **The platform provisions datasources only.** Every dashboard here is generated + from code with the Grafana Foundation SDK and shipped through the Ansible role, + so `ansible-playbook` rebuilds them from scratch. **Never save a dashboard from + the Grafana UI** — it won't survive the next run. + +Set `OTEL_EXPORTER_OTLP_ENDPOINT` (and friends) in `.env` to point at the Alloy +host. Nothing observability-related is hardcoded. + +--- + +## The five-minute demo + +Rehearse this before showing anyone. + +1. Delete every dashboard in the Grafana UI, then re-run + `ansible-playbook playbooks/grafana-stack.yml` — platform and dashboards both + rebuilt from code, one command. +2. Submit one transaction from the frontend. Follow that single request through + Grafana: RED dashboard → latency spike → exemplar → trace → the slow span → + that span's logs → back to the trace. One request, three signals, no + searching. +3. `make chaos-nightmare && make load-stress` — watch p95 diverge from p50, the + service map turn red at a specific node, the error budget burn, and the + fast-burn alert fire. +4. Walk through [`docs/INVESTIGATION.md`](docs/INVESTIGATION.md) — how the + pathologies were found from telemetry alone. + +--- + +## Repository layout + +``` +├─ CLAUDE.md # instructions for AI assistants — read before contributing +├─ Makefile +├─ docker-compose.yml +├─ Dockerfile # one image, all Python services +├─ db/ +│ ├─ schema.sql +│ ├─ optional_indexes.sql # the deliberately-omitted indexes +│ └─ seed_manifest.json # written by the seeder; the tests' source of truth +├─ scripts/ # seed.py, smoke.py +├─ common/ # settings, db, redis, http, logging, telemetry +├─ services/ # gateway ledger fraud sanctions insights notifier audit +│ └─ */chaos.py # each service's pathologies, clearly marked +├─ tests/ # black-box integration suite — a frozen contract +│ ├─ harness/ # load driver, pg_stat helpers, stream + container probes +│ └─ README.md # methodology, markers, known-flaky list +├─ observability/ # dashboard generator (Foundation SDK); output ships via the Ansible role +├─ loadtest/ # k6 scripts +└─ docs/ + ├─ roadmap/ # the ten-phase learning plan, one file per phase + │ ├─ intro.md # platform background, resource spine (read first) + │ ├─ phase-0.md … phase-10.md # one phase per file — status, message, concepts, build + │ └─ summary.md # one-page status/sequencing overview of all phases + ├─ phase-0a-test-suite-spec.md # spec the test suite was written from + ├─ phase-0-base-project-spec.md # spec the base app was built from + ├─ PATHOLOGIES.md # ⚠️ spoilers — the answer key + ├─ SLO.md # SLIs, SLOs, SLA, RTO/RPO, error budget policy + ├─ CONCEPTS.md # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words + ├─ INVESTIGATION.md # the Phase 8 bottleneck hunt, written as an incident doc + └─ PIPELINE-CHOICES.md # Filebeat vs Logstash vs OpenTelemetry — when and why +``` + +--- + +## Status + +- [ ] **Phase 0a** — integration test suite, written first, red against nothing +- [ ] **Phase 0b** — base project implemented until the suite is green +- [ ] **Phase 1** — concepts, first trace end to end +- [ ] **Phase 2** — distributed tracing across all seven services +- [ ] **Phase 3** — structured logs, correlated with traces +- [ ] **Phase 4** — metrics, RED, service map, exemplars +- [ ] **Phase 5** — dashboards and alerts as code +- [ ] **Phase 6** — k6 load testing, percentiles +- [ ] **Phase 7** — SLIs, SLOs, error budgets, RTO/RPO +- [ ] **Phase 8** — bottleneck hunt +- [ ] **Phase 9** — Filebeat / Logstash / OpenTelemetry judgment +- [ ] **Phase 10** — delivery + +Full plan: [`docs/roadmap/`](docs/roadmap/) — start with +[`intro.md`](docs/roadmap/intro.md), then the phase you're on, or +[`summary.md`](docs/roadmap/summary.md) for the one-page overview. + +--- + +## Non-goals + +No authentication, no authorization, no input hardening, no unit tests, no +Kubernetes, no production-grade storage. Ledgerline runs on a laptop under +Docker Compose and is never exposed to a network you don't control. + +The integration suite is deliberately the *only* test layer: it exists to pin +down the system's observable behaviour — working and broken — not to verify the +fake business logic. + +Things deliberately left for a later iteration, and worth naming when discussing +the project: tail-based sampling instead of 100%, Mimir instead of raw +Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII +redaction in the Collector, and multi-tenancy. diff --git a/docs/PATHOLOGIES.md b/docs/PATHOLOGIES.md new file mode 100644 index 0000000..a34d5a2 --- /dev/null +++ b/docs/PATHOLOGIES.md @@ -0,0 +1,156 @@ +# Pathology catalogue + +Thirteen deliberate pathologies, toggled at runtime via the `chaos:flags` Redis +hash (never a restart). Each lives in a `chaos.py` module inside its owning +service. All flags default off. + +> Spoiler note (see root `CLAUDE.md`): this file is a complete answer key for +> the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you, +> until that phase is done. + +--- + +## slow_sanctions (`sanctions`) + +**Mechanism:** `services/sanctions/chaos.py`. On every `/internal/lookup` +call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of +calls instead sleep a flat 5s -- longer than the 3s downstream timeout every +caller uses (`common/http.py`), so these surface as timeouts one hop up, in +`fraud`. + +**Symptom:** heavy-tailed latency on `POST /api/transactions`; most requests +look normal, a fat p95/p99 tail; occasional 504s from `fraud`/`gateway` with +no corresponding 5xx logged in `sanctions` itself (it never errors -- it's +just slow). + +## sanctions_errors (`sanctions`) + +**Mechanism:** `services/sanctions/chaos.py`. 3% of `/internal/lookup` calls +return HTTP 503 outright, chosen independently per call via `random.random()`. + +**Symptom:** a small, steady rate of 502/504 responses from `gateway`, fully +uncorrelated with load or account. `sanctions` logs a "lookup" line for every +attempt regardless, so its request count won't match its success count. + +## n_plus_one (`insights`) + +**Mechanism:** `services/insights/analytics.py::_n_plus_one_summary`. Instead +of one `LEFT JOIN` against `merchant_categories`, fetches up to 500 raw +transaction rows and issues one additional `SELECT ... FROM merchant_categories +WHERE id = :id` per row. + +**Symptom:** `GET /api/accounts/{id}/insights` for a high-volume (hot) account +issues 50-500+ SQL statements instead of 1-2, and its latency grows with the +account's transaction count rather than staying flat. + +## pool_starvation (`ledger`) + +**Mechanism:** `services/ledger/db.py`. Two engines are built at startup: a +normal one (`pool_size=20, max_overflow=10`) and a starved one (`pool_size=2, +max_overflow=0, pool_timeout=10`). Every `ledger` request picks which +already-running pool to borrow a connection from based on the current flag -- +this is what makes the toggle live without a restart, since SQLAlchemy pools +can't be resized in place. + +**Symptom:** throughput plateaus as concurrency increases past ~2 in-flight +requests; latency and queueing rise sharply under concurrent load while a +single request in isolation still looks fine. + +## row_lock_contention (`ledger`) + +**Mechanism:** `services/ledger/repository.py::settle_transaction`. Only for +accounts with `is_hot = true`: takes `SELECT balance_minor FROM accounts ... +FOR UPDATE` and holds the transaction open for an extra 150ms before +committing the balance update. + +**Symptom:** concurrent transactions against a hot account show p95 latency +several times higher than the same concurrency spread across many cold +accounts; cold-account traffic is unaffected. + +## blocking_event_loop (`fraud`) + +**Mechanism:** `services/fraud/chaos.py::synchronous_cpu_work`. Runs ~120ms of +synchronous SHA-256 hashing directly inside the `async def` handler for +`/internal/score` -- no `run_in_executor`/`to_thread`. + +**Symptom:** concurrent requests to `fraud` (and anything waiting on it) +serialise: wall-clock time for N concurrent requests approaches N × single-request +time instead of staying close to single-request time, even though nothing is +CPU-bound at the infrastructure level. + +## retry_storm (`gateway`) + +**Mechanism:** `services/gateway/orchestrator.py::_score_with_retry`. On any +non-2xx from `fraud`, retries immediately up to 3 more times -- no backoff, +no jitter, no circuit breaker. + +**Symptom:** when `sanctions` is also failing or slow, the number of +`sanctions` "lookup" log lines per transaction multiplies (up to 4x), and +downstream load spikes exactly when the downstream is already unhealthy -- +the opposite of what you want. + +## no_batching (`gateway`) + +**Mechanism:** `services/gateway/orchestrator.py::create_transaction`. Splits +the transaction into 1-5 synthetic "line items" and calls `fraud` once per +item, sequentially, instead of once for the whole transaction. + +**Symptom:** `sanctions` lookups per transaction multiply (roughly 2-5x) and +end-to-end latency for `POST /api/transactions` grows proportionally, even +though sanctions/fraud themselves are healthy. + +## unbounded_cache (`insights`) + +**Mechanism:** `services/insights/chaos.py` / `app.py`. Instead of the normal +Redis cache (keyed by account + current chaos-flag fingerprint, TTL below), +memoizes summaries in a module-level dict keyed by `(account_id, fingerprint, +time.time_ns())` -- a key that is different on every single call. It never +hits, and it never evicts. + +**Symptom:** `insights`' resident memory grows monotonically under repeated +identical-looking requests and never plateaus or drops; cache hit rate (if you +were watching Redis) would be zero -- the "cache" provides no speedup, only +a leak. + +## slow_consumer (`notifier`) + +**Mechanism:** `services/notifier/consumer.py`. Sleeps 250ms per stream +message before acking, inside the `notifier` consumer-group loop. + +**Symptom:** the `notifier` consumer-group lag on `stream:transactions` grows +under sustained load and drains slowly afterward, while the `audit` group +(same stream, independent group) keeps up fine -- isolating the problem to +one specific consumer, not the stream or producer. + +## cold_cache (`insights`) + +**Mechanism:** `services/insights/app.py`. The Redis summary cache TTL drops +from 900s to 60s. + +**Symptom:** repeated identical requests for the same account show periodic +latency spikes roughly every 60s (a full recompute) instead of one initial +spike followed by uniformly fast cache hits. + +## poison_data (`ledger`) + +**Mechanism:** `services/ledger/app.py` / `repository.py::find_null_currency_row`. +Every `POST /internal/transactions` checks whether the account has any +existing transaction with a `NULL` currency (one of the seeded malformed row +varieties, §8). When the flag is on, that row's `currency` is accessed with no +null guard (`.upper()` on `None`), which raises and surfaces as an unhandled +HTTP 500. When off, the same check is guarded and does nothing. + +**Symptom:** a specific, small set of accounts reliably 500 on every +transaction attempt while the flag is on, and behave completely normally +while it's off -- input data that was always there, suddenly fatal. + +## chatty_health (`gateway`) + +**Mechanism:** `services/gateway/orchestrator.py::_run_chatty_health_probe`. +Before doing any real work, `POST /api/transactions` calls `GET /healthz` +sequentially on all five downstreams, tagged with an `X-Ledgerline-Probe` +header so the extra hits are distinguishable from routine health polling. + +**Symptom:** every transaction request produces five extra, sequential +health-check round trips before the real work even starts, inflating latency +by a fixed per-request tax that has nothing to do with load or data size. diff --git a/docs/phase-0-base-project-spec.md b/docs/phase-0-base-project-spec.md new file mode 100644 index 0000000..adbabad --- /dev/null +++ b/docs/phase-0-base-project-spec.md @@ -0,0 +1,373 @@ +# Phase 0 — Technical Spec: "Ledgerline" base project + +> **How to use this file:** paste the whole thing into your coding agent as the task +> spec. It is written to be executed without further clarification. Read +> §2 (Non-goals) yourself before you start — the deliberate omissions are the +> point of the exercise. + +--- + +## 1. Mission + +Build **Ledgerline**, a deliberately imperfect microservice system that simulates a +small payments/finance backend. It exists for one reason: to be a realistic subject +for observability work in later phases. The business logic is fake and does not +need to be correct, defensible, or secure. What it *does* need to be is +**structurally realistic** — real network hops, real database queries, real async +message passing, real contention, real latency distributions, and real failure modes. + +The system must be runnable with `make up && make seed && make smoke` on a laptop. + +--- + +## 2. Non-goals — read carefully + +These are **hard constraints**, not preferences. Violating them destroys the value +of the project. + +- **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client, + no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no + APM agent, no Sentry. Later phases add all of this by hand; that is the lesson. +- **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add + indexes, connection-pool tuning, batching, backoff, or caching beyond what is + specified. +- **DO NOT add authentication, authorization, rate limiting, or input hardening.** + A single hardcoded demo user is fine. +- **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the + deliverable. +- **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only. +- **DO NOT abstract prematurely.** Duplication across services is acceptable and + preferred over a clever shared framework. + +The only logging permitted is Python's standard `logging` module with a plain-text +formatter (§9). + +--- + +## 3. Technology + +| Concern | Choice | +|---|---| +| Language | Python 3.12 | +| Web framework | FastAPI + `uvicorn` (one worker per service) | +| HTTP client | `httpx` (async) | +| DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` | +| Cache / queue | Redis 7 (`redis-py` asyncio client) | +| Async messaging | Redis **Streams** (not Celery, not RabbitMQ) | +| Validation | Pydantic v2 | +| Packaging | One shared `Dockerfile`, one image, `CMD` differs per service | +| Orchestration | Docker Compose | +| Schema | A single idempotent `db/schema.sql` — no Alembic | +| Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework | + +Redis Streams is chosen deliberately: it has **no automatic context propagation**, +so a later phase has to implement it manually. Do not swap it for something with +built-in instrumentation. + +--- + +## 4. Services + +Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`. + +| # | Service | Port | Role | Stores | +|---|---|---|---|---| +| 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) | +| 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres | +| 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) | +| 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) | +| 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis | +| 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres | +| 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres | + +Every HTTP service exposes: + +- `GET /healthz` → `{"status": "ok", "service": ""}` — liveness, no dependency checks +- `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable +- `GET /` → `{"service": "", "version": "0.1.0"}` + +--- + +## 5. Domain model (`db/schema.sql`) + +``` +accounts (id UUID pk, holder_name text, currency char(3), + balance_minor bigint, is_hot boolean default false, + created_at timestamptz) + +merchant_categories (id smallint pk, code text unique, label text, + risk_weight numeric(3,2)) + +transactions (id UUID pk, account_id UUID, kind text, -- deposit|withdrawal|purchase + amount_minor bigint, currency char(3), + merchant_name text, category_id smallint, + status text, -- pending|settled|rejected + risk_score numeric(4,3), created_at timestamptz) + +sanctions_entries (id serial pk, normalized_name text, country char(2), + list_source text, added_at timestamptz) + +notifications (id UUID pk, transaction_id UUID, channel text, -- email|push|sms + status text, attempts int, sent_at timestamptz) + +audit_events (id bigserial pk, transaction_id UUID, event_type text, + payload jsonb, recorded_at timestamptz) +``` + +**Index policy — important.** Create only these indexes: + +- primary keys (implicit) +- `accounts(holder_name)` +- `sanctions_entries(normalized_name)` + +Deliberately **omit** `transactions(account_id, created_at)` and +`transactions(category_id)`. Put the `CREATE INDEX` statements for those in +`db/optional_indexes.sql`, applied only by `make fix-index`. + +--- + +## 6. Request flows + +### 6.1 `POST /api/transactions` — the primary flow + +The trace this produces should be ~4 levels deep with one async fork. + +``` +browser + └─ gateway POST /api/transactions + ├─ ledger POST /internal/transactions (create as `pending`, row lock on account) + ├─ fraud POST /internal/score (risk scoring) + │ └─ sanctions POST /internal/lookup (watchlist check — slow vendor) + ├─ ledger POST /internal/transactions/{id}/settle (or /reject if risk_score > 0.8) + ├─ redis XADD stream:transactions (fire-and-forget) + │ ├─ notifier consumer group `notifier` + │ └─ audit consumer group `audit` + └─ insights GET /internal/accounts/{id}/summary (cached) +``` + +Response body: + +```json +{ + "transaction_id": "…", "status": "settled", "risk_score": 0.12, + "balance_minor": 481200, "summary": { "…": "…" }, + "request_id": "…", "elapsed_ms": 843 +} +``` + +### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow + +`gateway → insights`. `insights` runs a 120-day aggregation over `transactions`. +This is where the N+1 and the missing index bite. + +### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown + +### 6.4 Admin + +- `GET /api/admin/chaos` → current flag values +- `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}` +- `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}` + +Flags live in a Redis hash `chaos:flags` so every service reads the same state +without a restart. Each service reads the hash on **every request** (a tiny bit of +extra Redis load is fine and realistic). + +--- + +## 7. The pathology catalogue + +Each pathology lives in a clearly named `chaos.py` module inside its owning service, +with a docstring naming it. All flags default **off**. + +| Flag | Service | Behaviour when enabled | +|---|---|---| +| `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). | +| `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. | +| `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. | +| `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) | +| `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. | +| `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. | +| `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. | +| `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. | +| `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. | +| `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. | +| `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. | +| `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. | +| `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. | + +**Profiles** + +- `clean` — all off. +- `light` — `slow_sanctions`, `cold_cache`. +- `nightmare` — all thirteen on. +- `random` — pick a random subset of size 3–6, using a seed derived from + `time.time()` and **not logged anywhere**. Store the active set in Redis under + `chaos:flags` only, never write it to a file or print it. (This makes the later + bottleneck-hunting exercise a genuine hunt.) + +Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the +symptom a reader should expect. Do **not** reproduce or summarise its contents in +your final chat response. + +--- + +## 8. Seed data (`scripts/seed.py`) + +Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert. + +- **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD, + 10% GBP. Balances log-normal, median ≈ 250 000 minor units. + Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets. +- **45 merchant categories** with `risk_weight` between 0.01 and 0.95. +- **250 000 transactions** spread over the last 120 days. + - Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip. + - Amounts log-normal. + - Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest + on a Zipf-ish distribution. + - `status` 92% `settled`, 5% `rejected`, 3% `pending`. +- **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account + holders, so a meaningful fraction of requests genuinely score as high risk. +- **~2% malformed rows** distributed through `transactions`: + - `currency` = `NULL` + - negative `amount_minor` on a `deposit` + - `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence + - `created_at` set 3 days in the future + - `category_id` pointing at a non-existent category + +Print a summary table at the end (row counts per table, hot account IDs, count of +malformed rows). Seeding 250k rows must complete in under 60 seconds — use +`COPY` / `executemany`, not per-row inserts. + +--- + +## 9. Logging (the deliberately bad baseline) + +Every service configures: + +```python +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s %(message)s", +) +``` + +- Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value. +- `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it + as a header on every downstream `httpx` call. Each service logs it manually at the + start of each handler. **This is the only correlation mechanism that exists.** + It is intentionally weak — it does not survive the Redis Stream hop, it carries no + timing, and it cannot express parent/child relationships. +- Set `PYTHONUNBUFFERED=1` in the Dockerfile. +- Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK. + +--- + +## 10. Frontend + +Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`, +`app.js`, `style.css`. No bundler, no npm, no CDN dependencies. + +Layout, top to bottom: + +1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`), + amount input, merchant name input, kind radio (deposit / withdrawal / purchase), + Submit button. +2. **Result panel** — pretty-printed JSON response, plus the browser-measured + round-trip time in ms, plus the `request_id` shown in a monospace font that is + easy to copy. +3. **Load burst** — a button that fires *N* concurrent requests (N selectable: + 10 / 50 / 200) against random accounts, then renders a small table: + count, success, errors, min / mean / p50 / p95 / max ms, measured client-side. + This is a crude stand-in for a real load tool; a later phase replaces it with k6. +4. **Chaos panel** — a checkbox per flag reflecting current state from + `GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile + buttons (`clean`, `light`, `nightmare`, `random`). +5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s + via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green + or red dot per service. + +Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS. + +--- + +## 11. Repository layout + +``` +ledgerline/ +├─ Makefile +├─ docker-compose.yml +├─ Dockerfile # single image for all Python services +├─ pyproject.toml # or requirements.txt — one dependency set +├─ .env.example +├─ README.md +├─ db/ +│ ├─ schema.sql +│ └─ optional_indexes.sql +├─ scripts/ +│ ├─ seed.py +│ └─ smoke.py +├─ common/ +│ ├─ __init__.py +│ ├─ settings.py # pydantic-settings, env-driven +│ ├─ db.py # async engine/session factory +│ ├─ redis_client.py +│ ├─ http.py # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough +│ └─ logging_config.py # the plain-text logging setup from §9 +├─ services/ +│ ├─ gateway/ (app.py, chaos.py, orchestrator.py, static/) +│ ├─ ledger/ (app.py, chaos.py, repository.py) +│ ├─ fraud/ (app.py, chaos.py, scoring.py) +│ ├─ sanctions/ (app.py, chaos.py, repository.py) +│ ├─ insights/ (app.py, chaos.py, analytics.py) +│ ├─ notifier/ (app.py, chaos.py, consumer.py) +│ └─ audit/ (worker.py, consumer.py) +└─ docs/ + └─ PATHOLOGIES.md +``` + +`docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use +`depends_on` with `condition: service_healthy` for postgres and redis. Mount the +source read-write and run uvicorn with `--reload` so iteration is fast. + +Reserve — but leave empty and unreferenced — a top-level `observability/` directory +with a `.gitkeep`. Later phases fill it. + +--- + +## 12. Makefile targets + +| Target | Effect | +|---|---| +| `make up` / `make down` | compose up -d / down | +| `make logs` | `docker compose logs -f` | +| `make seed` | apply `db/schema.sql`, run `scripts/seed.py` | +| `make reset` | `down -v`, `up`, `seed` | +| `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s | +| `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile | +| `make fix-index` | apply `db/optional_indexes.sql` | +| `make drop-index` | drop them again | + +--- + +## 13. Acceptance criteria + +The task is done when all of these hold on a clean checkout: + +1. `make reset` completes in under 3 minutes and prints the seed summary table. +2. `make smoke` passes with `chaos-clean`. +3. The frontend loads at `http://localhost:8000`, the account dropdown is populated, + and a submitted transaction returns a settled or rejected result. +4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under + ~250 ms and zero errors. +5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error + rate between 2% and 10% — degraded but not fully broken. +6. Toggling any single flag via the chaos panel changes behaviour **without a + restart**. +7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`. +8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns + nothing. +9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies. + +In your final message, report only: services started, row counts seeded, the +p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you +deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`. diff --git a/docs/phase-0a-test-suite-spec.md b/docs/phase-0a-test-suite-spec.md new file mode 100644 index 0000000..928c6fb --- /dev/null +++ b/docs/phase-0a-test-suite-spec.md @@ -0,0 +1,508 @@ +# Phase 0a — Technical Spec: Ledgerline integration test suite + +> **How to use this file:** paste it into the **test-authoring agent** together +> with `phase-0-base-project-spec.md`. That agent writes only `tests/`. +> A second agent then implements the services against the resulting suite. +> See §1 for the workflow contract between them. + +--- + +## 1. The two-agent workflow + +### Agent A — test author (this spec) + +- Reads `phase-0-base-project-spec.md` (the system contract) and this file. +- Creates **only** these paths: + ``` + tests/** pytest.ini (or pyproject [tool.pytest]) + requirements-test.txt + tests/README.md + tests/RED-BASELINE.md + ``` + Plus the `test-*` targets appended to the `Makefile`. +- Creates **nothing** under `services/`, `common/`, `db/`, `scripts/`, + `observability/` or `loadtest/`. Not even a stub. If a test needs a fixture + from the system, it asserts against the contract in §4 rather than importing + application code. +- **Imports nothing from the application.** Tests talk to the system over HTTP, + Postgres, Redis, and the Docker CLI. No `from services.gateway import …`. + This is what keeps them behavioural rather than implementation-coupled. +- Runs the suite against an empty repo and confirms it is **red**, then writes + `tests/RED-BASELINE.md` listing, per test, *why* it currently fails + (connection refused / 404 / assertion / missing table). A test that passes + against nothing is a broken test — find it and fix it. + +### Agent B — implementer + +- Reads `phase-0-base-project-spec.md` and treats `tests/` as a **read-only + contract**. +- **Must not modify, delete, skip, xfail, or loosen any test**, or change + `pytest.ini`, markers, thresholds, or fixtures. If a test appears wrong, + impossible, or contradictory: **stop and report it**, don't edit it. +- Must implement each pathology by the mechanism described in + `phase-0-base-project-spec.md` §7 — not as a generic `sleep()` tuned to make a + timing assertion pass. Several assertions in §6 are deliberately built so that + faking is harder than implementing (query counts, `EXPLAIN` plan shapes, Redis + consumer lag, process RSS). Treat that as the intent everywhere, including + where the assertion happens to be simpler. +- Done when `make test` is green three consecutive runs. + +--- + +## 2. Design principles for this suite + +These are the rules the tests must obey. They matter more than any individual +test. + +1. **Assert goals, not implementation.** No test may reference a function name, + module path, class, SQL string, log format, or internal attribute. A test may + reference the *public contract*: HTTP routes, status codes, JSON field names, + table names, Redis stream and consumer-group names, chaos flag names. +2. **Prefer assertions that are hard to satisfy without the real mechanism.** + Wall-clock latency alone is fakeable with `sleep()`. Query counts from + `pg_stat_statements`, `EXPLAIN` plan node types, Redis consumer-group lag, + container RSS, and log-line counts on a *different service* are not. Where + both are available, assert on both. +3. **Measure differentially, never absolutely.** With one exception (§5.7), + every pathology test compares flag-on against flag-off in the same run, on + the same machine, and asserts a *ratio*. Absolute millisecond thresholds do + not survive a different laptop. +4. **Design for a noisy laptop.** Generous margins, warm-up requests discarded, + median-of-N measurement runs. A test that fails when Slack is open is worse + than no test. +5. **Fast tests and slow tests are separate.** `make test-contract` must run in + under 60 seconds. Pathology tests may take minutes and live behind their own + marker. +6. **Every test restores state.** A test that enables a chaos flag must reset + the profile in teardown, even on failure. + +--- + +## 3. Stack, layout, and execution + +Tests run **on the host**, against the Compose stack on `localhost`. They do not +run inside a container and do not use testcontainers — the stack is started by +`make up` beforehand. + +``` +tests/ +├─ README.md +├─ RED-BASELINE.md +├─ conftest.py # fixtures: clients, chaos control, seed manifest +├─ harness/ +│ ├─ load.py # concurrent request driver + LatencyReport +│ ├─ pg.py # pg_stat_statements deltas, EXPLAIN helpers +│ ├─ streams.py # Redis XINFO GROUPS helpers +│ └─ containers.py # docker stats / docker logs helpers +├─ test_contract_health.py +├─ test_contract_transactions.py +├─ test_contract_insights.py +├─ test_contract_async.py +├─ test_contract_chaos.py +├─ test_contract_frontend.py +├─ test_seed_data.py +├─ test_baseline_health.py +├─ test_pathologies.py +├─ test_nightmare.py +└─ test_no_observability.py +``` + +Dependencies (`requirements-test.txt`): `pytest`, `pytest-asyncio`, +`pytest-timeout`, `httpx`, `psycopg[binary]`, `redis`. Docker access via +`subprocess` calling the `docker` CLI — no SDK. + +### Markers + +| Marker | Runtime | Contents | +|---|---|---| +| `contract` | < 60 s | Functional behaviour, chaos-clean | +| `seed` | < 20 s | Seed data integrity | +| `baseline` | < 90 s | The clean system is genuinely healthy | +| `pathology` | 5–12 min | One test per pathology, differential | +| `slow` | — | Additional marker on the soak-ish ones (`unbounded_cache`, `cold_cache`) | +| `guard` | < 5 s | The "no observability yet" assertions | + +Configure `pytest.ini` with `--strict-markers`, `asyncio_mode = auto`, and a +global `timeout = 300`. + +### Make targets to append + +| Target | Command | +|---|---| +| `test` | everything except `slow` | +| `test-contract` | `-m "contract or seed or guard"` | +| `test-baseline` | `-m baseline` | +| `test-pathology` | `-m "pathology and not slow"` | +| `test-slow` | `-m slow` | +| `test-all` | no marker filter | + +--- + +## 4. Contract additions required for testability + +These extend `phase-0-base-project-spec.md`. Agent B must implement them; Agent A +may assume them. + +### 4.1 Seed manifest + +`scripts/seed.py` writes `db/seed_manifest.json` **and** a `seed_manifest` table +(single row, `data jsonb`) with the same content. Tests read the table, so they +don't depend on a host file path. + +```json +{ + "seed": 42, + "generated_at": "…", + "counts": { "accounts": 800, "transactions": 250000, + "merchant_categories": 45, "sanctions_entries": 12000, + "malformed_transactions": 5013 }, + "hot_account_ids": ["…", "…", "…", "…", "…"], + "cold_account_ids_sample": ["…"], // 50 non-hot accounts, for spread tests + "sanctioned_account_ids": ["…"], // accounts whose holder matches the watchlist + "poison_account_ids": ["…"], // accounts owning malformed rows + "clean_account_id": "…" // no malformed rows, not hot, not sanctioned +} +``` + +### 4.2 Response envelope + +`POST /api/transactions` → **201** on success: + +```json +{ "transaction_id": "", "status": "settled|rejected", + "risk_score": 0.0-1.0, "balance_minor": 0, + "summary": { }, "request_id": "", "elapsed_ms": 0 } +``` + +Errors return a consistent envelope with the appropriate status: + +```json +{ "error": "", "detail": "", + "request_id": "" } +``` + +Status mapping: `400` invalid input, `404` unknown account, `422` validation, +`502` downstream returned an error, `504` downstream timed out, `500` +unhandled — which is exactly what `poison_data` produces, and what §6.12 asserts. + +### 4.3 Chaos API + +- `GET /api/admin/chaos` → `{"profile": "", "flags": {"": bool, …}}` + containing **all thirteen** flags, always. +- `POST /api/admin/chaos` → `{"flag": "", "enabled": bool}`; returns the + full new state. Unknown flag → `400`. +- `POST /api/admin/chaos/profile` → `{"profile": "clean|light|nightmare|random"}`; + returns the full new state. For `random`, the response reports the flags — + **but `make chaos-random` must not print them**, and the seed is never logged. +- `GET /api/health/all` → `{"services": {"": "ok"|"unhealthy", …}}` for all + six HTTP services. + +### 4.4 `pg_stat_statements` + +The Postgres image must load `pg_stat_statements` (`shared_preload_libraries`) +and the extension must be created by `db/schema.sql`. Tests use +`pg_stat_statements_reset()` around a single request to count queries. + +### 4.5 Deterministic seeding + +`SEED` env var, default `42`. Same seed ⇒ identical manifest, identical row +counts, identical hot/poison account IDs. + +--- + +## 5. Goal-to-test traceability + +Every test must map to one of these goals. Put the goal ID in the test's +docstring. This is what keeps the suite about intent rather than mechanics. + +| ID | Goal | Where | +|---|---|---| +| **G1** | A transaction traverses all seven services and returns a coherent result | §5.1 | +| **G2** | The system is genuinely multi-service, not a collapsed monolith | §5.2 | +| **G3** | Async work happens after the response, via two independent consumer groups | §5.3 | +| **G4** | The seed data is large, skewed, and contains the intended poison rows | §5.4 | +| **G5** | Chaos flags toggle at runtime without a restart | §5.5 | +| **G6** | Each pathology measurably degrades the system when enabled | §6 | +| **G7** | The clean baseline is actually healthy — otherwise G6 means nothing | §5.6 | +| **G8** | Nightmare mode degrades but does not collapse | §5.7 | +| **G9** | Failure is contained: timeouts fire, the gateway doesn't wedge | §5.8 | +| **G10** | No observability tooling is present yet | §5.9 | +| **G11** | The frontend and its supporting endpoints are served | §5.10 | + +### 5.1 G1 — end-to-end transaction (`contract`) + +- Submitting a valid transaction for `clean_account_id` returns 201 with the full + envelope; `transaction_id` is a UUID; `status` ∈ {settled, rejected}; + `risk_score` ∈ [0,1]; `elapsed_ms` > 0. +- The transaction is durable: a row with that ID exists in `transactions` with a + terminal status within 2 s. +- The balance moves in the correct direction for a deposit and for a withdrawal, + by exactly `amount_minor`, when the transaction settles. +- A transaction for a `sanctioned_account_ids` holder yields a materially higher + `risk_score` than one for `clean_account_id` (assert the ordering, not a value). +- `GET /api/accounts/{id}/insights` returns a summary for a seeded account. +- Unknown account → 404 with the error envelope. Negative amount on a deposit → + 400 or 422. Missing required field → 422. + +### 5.2 G2 — really seven services (`contract`) + +- All six HTTP services answer `/healthz` on their own port with their own name. +- `/readyz` returns 200 for all six. +- `GET /api/health/all` reports all six as ok. +- Stopping `sanctions` (`docker compose stop sanctions`) causes + `GET /api/health/all` to report it unhealthy within 15 s, and + `POST /api/transactions` to fail with 502 or 504 — **not** 200. Restart it in + teardown and wait for ready. + *(This is the test that catches a collapsed monolith: if transactions still + succeed with `sanctions` stopped, the service isn't really in the path.)* + +### 5.3 G3 — asynchronous work (`contract`) + +- After a successful transaction, a row appears in `notifications` **and** in + `audit_events` for that `transaction_id` within 10 s. +- Both appear — proving two independent consumer groups, not one consumer doing + both jobs. +- `XINFO GROUPS stream:transactions` reports exactly two groups, named + `notifier` and `audit`. +- The API response returns **before** the notification row exists at least once + across 20 attempts, proving the work is genuinely off the request path. + *(Sample the DB immediately on response; assert at least one miss. Tolerant of + a fast machine winning the race sometimes — hence 20 attempts, not 1.)* + +### 5.4 G4 — seed data (`seed`) + +Against the manifest and the database: + +- Row counts match the manifest and the spec's targets within ±2%. +- Exactly 5 accounts have `is_hot = true`, and they match `hot_account_ids`. +- The 5 hot accounts own between 20% and 40% of all transactions. +- Transactions span ≥ 110 distinct days. +- Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each). +- Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points). +- Malformed rows exist in all five described varieties: null currency, negative + amount on a deposit, non-ASCII merchant name, future `created_at`, orphan + `category_id`. Assert **at least one of each**, and 1%–4% overall. +- ≥ 30 sanctions entries match seeded holder names. +- Re-running the seed with the same `SEED` yields an identical manifest + (compare the `counts` and `hot_account_ids`). +- Neither `transactions(account_id, created_at)` nor `transactions(category_id)` + exists in `pg_indexes` after a fresh seed. + +### 5.5 G5 — chaos control (`contract`) + +- `GET /api/admin/chaos` lists all thirteen flags by name. +- Toggling a single flag changes only that flag. +- Each profile sets the expected shape: `clean` → all false; `nightmare` → all + true; `light` → exactly the two documented flags. +- **No restart:** with `chaos-clean` set, record the `insights` latency; enable + `n_plus_one` via the API; without restarting any container, the query count for + the same request rises. Assert on the query-count delta (§6.3's helper), so + this test proves live reconfiguration rather than timing noise. +- Unknown flag name → 400. +- Teardown restores `clean`. + +### 5.6 G7 — the clean baseline is healthy (`baseline`) + +With `chaos-clean`, after 20 discarded warm-up requests, drive 200 requests at +concurrency 10 against random cold accounts: + +- error rate == 0 +- p95 < 400 ms *(the one absolute threshold in the functional set; if it fails on + a slow machine, that machine cannot run the pathology suite meaningfully + either, and the failure message must say so)* +- p99 / p50 < 5 — the clean system has no fat tail +- `insights` for a cold account issues fewer than 10 SQL statements +- Redis consumer lag returns to 0 within 15 s of the burst finishing + +### 5.7 G8 — nightmare degrades but survives (`pathology`) + +Same driver, `chaos-clean` then `chaos-nightmare`, 300 requests at concurrency 20 +each: + +- nightmare p95 ≥ **5×** clean p95 +- nightmare error rate between **2% and 15%** — degraded, not dead +- at least 80% of requests still return a valid envelope (either success or a + structured error, never a connection reset or an empty body) +- `GET /api/health/all` still responds within 5 s during the burst + +### 5.8 G9 — failure containment (`contract`) + +- With `sanctions_errors` at its maximum and `retry_storm` off, no request takes + longer than 10 s. The 3 s downstream timeout must actually fire. +- With `sanctions` stopped entirely, `POST /api/transactions` returns 502/504 + within 10 s, and the gateway still answers `/healthz` in under 1 s throughout. +- No request ever returns a 200 with a `null` `transaction_id`. + +### 5.9 G10 — no observability yet (`guard`) + +Static and runtime assertions that the Phase 0 "before" state is intact: + +- No file under `services/` or `common/` imports `opentelemetry`, + `prometheus_client`, `structlog`, `sentry_sdk`, or `elasticapm`. +- Neither `requirements*.txt` nor `pyproject.toml` lists them (excluding + `requirements-test.txt`). +- No service exposes `/metrics` (assert 404 on all six). +- `docker compose logs gateway` emits plain text, not JSON: sample 20 lines and + assert none parse as a JSON object. + +> This test class is **expected to be deleted at the start of Phase 2**, not +> maintained. Say so in its module docstring. + +### 5.10 G11 — frontend (`contract`) + +- `GET /` returns 200, `text/html`, and references `app.js` and `style.css`. +- Those two assets return 200 with sensible content types. +- `GET /api/accounts` returns ≥ 100 accounts, each with `id`, `holder_name`, + `currency`, `balance_minor`. + +No browser automation. Serving and shape only. + +--- + +## 6. G6 — pathology tests + +**One test per flag.** Each follows the same skeleton: + +``` +1. set chaos-clean +2. warm up (discard) +3. measure → baseline +4. enable exactly one flag via the API +5. warm up (discard) +6. measure → degraded +7. assert the ratio, with the failure message printing both measurements +8. teardown: chaos-clean +``` + +Measurement runs the median of 3 for latency-based assertions. Failure messages +must always include both raw numbers — a bare `assert ratio > 3` is useless at +2 a.m. + +| § | Flag | Primary signal (hard to fake) | Secondary signal | Assertion | +|---|---|---|---|---| +| 6.1 | `slow_sanctions` | — | p99/p50 of `POST /api/transactions` | degraded p99/p50 ≥ **6**, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99 | +| 6.2 | `sanctions_errors` | count of 502/504 responses over 300 requests | — | degraded error rate ∈ [1%, 10%]; baseline == 0 | +| 6.3 | `n_plus_one` | `pg_stat_statements` call delta across **one** insights request | latency ratio | degraded ≥ **50** statements; baseline ≤ 10; latency ratio ≥ 2 | +| 6.4 | *missing index* (`make drop-index` / `fix-index`, not a chaos flag) | `EXPLAIN (ANALYZE, FORMAT JSON)` on the insights aggregation contains a `Seq Scan` on `transactions`; after `fix-index` it contains an `Index Scan`/`Bitmap Index Scan` | latency ratio | plan node types as stated; unindexed latency ≥ **3×** indexed. Teardown restores the dropped state. | +| 6.5 | `pool_starvation` | throughput (req/s) at concurrency 5 vs 25 | p95 ratio | with the flag on, throughput at c=25 is ≤ **1.4×** throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25 | +| 6.6 | `row_lock_contention` | 25 concurrent transactions against **one hot account** vs 25 spread across 25 cold accounts, both with the flag on | — | single-hot-account p95 ≥ **3×** spread p95; with the flag off the same comparison is ≤ 1.8× | +| 6.7 | `blocking_event_loop` | wall-clock for 10 concurrent requests ÷ wall-clock for 1 request | — | degraded ratio ≥ **6** (serialised); baseline ratio ≤ 2.5 (concurrent) | +| 6.8 | `retry_storm` | count of `sanctions` container log lines matching a lookup, per transaction, with `sanctions_errors` also on | — | degraded ≥ **2.0** lookups/transaction; baseline (retry off, errors on) ≤ 1.2. *Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.* | +| 6.9 | `no_batching` | `sanctions` lookup log lines per transaction, `sanctions_errors` **off** | latency ratio | degraded ≥ **2.0** lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5 | +| 6.10 | `unbounded_cache` (`slow`) | `insights` container RSS via `docker stats` before vs after 2000 requests with **distinct** query parameters, sampled 30 s after the burst ends | — | degraded RSS growth ≥ **20 MB** and still elevated at +30 s; baseline growth ≤ 8 MB | +| 6.11 | `slow_consumer` | `XINFO GROUPS stream:transactions` → `lag`/`pending` for the `notifier` group after a 150-transaction burst | drain time | degraded lag ≥ **30** at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s | +| 6.12 | `cold_cache` (`slow`) | same insights request at 1 rps for 150 s; count samples > 3× the median | — | degraded ≥ **2** spikes (a 60 s TTL over 150 s); baseline ≤ 1 | +| 6.13 | `poison_data` | request against each `poison_account_ids` entry | — | degraded produces ≥ 1 response with status **500**; baseline produces zero 500s across the same requests (4xx or 200 only) | +| 6.14 | `chatty_health` | `/healthz` hits on downstream containers during a single transaction, counted from logs | — | degraded ≥ **5** healthz lines across downstreams per transaction; baseline == 0 | + +**Note on log-line counting (6.8, 6.9, 6.14).** At Phase 0 the plain-text log is +the only cross-service observability surface that exists — which is precisely the +deficiency this whole project sets out to fix. Implement these helpers behind +`harness/containers.py::count_log_lines(service, pattern, since)` so there is +exactly one place to change. **From Phase 3 these assertions should be rewritten +against structured logs, and from Phase 4 against span counts** — at which point +they become both simpler and more robust. Note this in `tests/README.md`; it is a +deliberate teaching artefact, not a shortcut. + +**Anti-fake note for Agent B.** 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied +by inserting a `sleep()`: they require, respectively, real repeated queries, a +real missing index, real unbounded allocation, real consumer slowness, and a real +unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12) +are stated as ratios for the same reason — passing them by faking is more work +than implementing the mechanism described in the Phase 0 spec. + +--- + +## 7. The measurement harness + +`harness/load.py` provides one primitive: + +```python +async def drive( + method: str, url_factory: Callable[[], tuple[str, dict]], + total: int, concurrency: int, + warmup: int = 20, timeout: float = 15.0, +) -> LatencyReport +``` + +`LatencyReport` carries `count, ok, errors, error_rate, status_counts, +p50, p95, p99, max, throughput_rps, wall_seconds`, plus a `__str__` suitable for +dropping straight into an assertion message. + +Rules: + +- Warm-up requests are issued and discarded before timing starts. +- Percentiles from the raw sorted sample, nearest-rank. No interpolation, no + averaging of percentiles across runs — **median of the p95s**, never the mean. +- A closed-loop driver (fixed concurrency) is used deliberately for its + simplicity here. Note in `tests/README.md` that this under-measures latency + under saturation (coordinated omission) and that Phase 6 replaces it with k6's + arrival-rate executors. Don't try to fix it in the test suite. +- `measure_median(n=3, fn)` helper for the pathology tests. + +`harness/pg.py`: + +```python +@contextmanager +def statement_count(dsn) -> Counter # resets pg_stat_statements, yields, returns delta +def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON) +def plan_node_types(plan: dict) -> set[str] +``` + +`harness/streams.py`: `group_lag(group)`, `wait_for_drain(group, timeout)`. + +`harness/containers.py`: `rss_mb(service)`, `count_log_lines(service, pattern, +since)`, `stop(service)`, `start(service)`, `wait_ready(service, timeout)`. + +--- + +## 8. Fixtures (`conftest.py`) + +| Fixture | Scope | Purpose | +|---|---|---| +| `stack_up` | session | Assert the Compose stack is running and all six services are ready; **fail fast with a clear message** ("run `make up && make seed` first") rather than timing out | +| `seeded` | session | Assert the seed manifest exists and counts are non-zero; fail fast otherwise | +| `manifest` | session | Parsed seed manifest | +| `api` | function | `httpx.AsyncClient` against the gateway, 15 s timeout | +| `dsn` | session | Postgres DSN from env | +| `redis_client` | session | — | +| `chaos` | function | `chaos.set_profile(name)`, `chaos.set_flag(name, bool)`, `chaos.get()`. **Autouse teardown resets to `clean`**, even on failure | +| `fresh_balances` | function | Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs | + +Every environment value (ports, DSN, Redis URL) comes from env with the Compose +defaults as fallbacks. No hardcoded values buried in test bodies. + +--- + +## 9. `tests/README.md` — what Agent A must write + +- How to run each marker set, and the prerequisite (`make up && make seed`). +- The measurement methodology and why ratios are used rather than absolutes. +- The known-flaky list: 6.10 and 6.12 are the least stable; document the retry + strategy and what a genuine failure looks like versus noise. +- The coordinated-omission caveat on the closed-loop driver. +- The Phase 3/4 migration note for the log-counting helpers. +- An explicit statement that `test_no_observability.py` is deleted at Phase 2. + +--- + +## 10. Acceptance criteria + +For **Agent A**: + +1. `pytest --collect-only` succeeds and every test carries exactly one goal ID in + its docstring and at least one marker. +2. `grep -rn "from services\|from common\|import services\|import common" tests/` + returns nothing. +3. Every §5 and §6 item has a corresponding test; `tests/README.md` contains the + goal-to-test table. +4. The suite is **red** against an empty repo, and `tests/RED-BASELINE.md` + records the failure reason per test. No test passes vacuously. +5. `make test-contract` completes (failing) in under 60 s. + +For **Agent B**: + +1. `make test` green three consecutive runs. +2. `make test-slow` green (may be run once). +3. `git diff --stat tests/` against Agent A's commit is empty. +4. `docs/PATHOLOGIES.md` written, and not summarised in the final response. diff --git a/docs/roadmap/intro.md b/docs/roadmap/intro.md new file mode 100644 index 0000000..1d4dd2c --- /dev/null +++ b/docs/roadmap/intro.md @@ -0,0 +1,377 @@ +# Observability & APM — Delivery Roadmap for Babak's Task + +*Part of `docs/roadmap/` — platform background and reading guidance shared by +every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`; +[`summary.md`](summary.md) gives the one-page overview and status of all of +them.* + +Ten phases. Each one: concepts first, then the build, then a checkable exit +condition. Total time-boxed at roughly **4–6 weeks part-time** (evenings + +weekends). Nothing here is theory for theory's sake — every concept block is +followed immediately by the place it lands in the code. + +--- + +## Read this first — the four questions blocking you + +### "What *is* an APM? Do you install it, use it, implement it, or build it?" + +APM (Application Performance Monitoring) is a **product category and a capability**, +not a protocol or a single tool. A system has APM when you can answer, for +production traffic: what's the request rate, error rate and latency distribution +per endpoint; which services call which; where did *this specific* request spend its +time; which database query or external call was slow; and what error was thrown, +with the stack. + +The verb depends on the vendor model: + +- **Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM).** You + *install* an agent that attaches to your runtime and ships to their backend. Mostly + a deployment exercise. Turnkey, expensive, proprietary. +- **Grafana stack — what Babak asked for.** There is no product called "Grafana APM" + that you install. You **assemble** the capability from four layers: + + | Layer | What you do | Component | + |---|---|---| + | Instrumentation | *Implement* — this is code you write | OpenTelemetry SDK inside your services | + | Pipeline | *Deploy + configure* | OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution) | + | Storage | *Deploy* | Tempo (traces), Loki (logs), Prometheus or Mimir (metrics) | + | Query & UI | *Configure* | Grafana + provisioned datasources with cross-links | + + So: you **implement** instrumentation, you **deploy** the pipeline and stores, you + **configure** the correlation. The "APM" is the emergent property of all four + working together. + +The distinction that actually matters: APM data is generated *inside* the +application, keyed to *individual requests*, and *correlated* across signals. +Infrastructure monitoring (CPU, disk, pod restarts) is generated *outside* the +application and aggregated. You need both; only the first is APM. + +### "What's the difference between Logs, Metrics, Events and Traces?" + +The clean way to hold this in your head is: **an event is the atom; the other three +are shapes you give it.** + +| | What it is | Cost shape | Answers | +|---|---|---|---| +| **Event** | A structured record of one thing that happened, at one instant, with arbitrary attributes attached | Grows with traffic × attribute count | "What exactly happened here?" | +| **Log** | An event rendered for storage/search. A *structured* log is literally an event; an unstructured log is an event you threw the schema away from | Grows with traffic | "Why did it fail?" | +| **Metric** | A pre-aggregated numeric measurement over many events, sliced by a small set of labels | Grows with *label cardinality*, ~flat in traffic | "Is it broken? How much? Since when?" | +| **Trace** | A causally linked tree of events that each have a *duration* and a parent — a span tree spanning process boundaries | Grows with traffic × span count | "Where did the time go, across services?" | + +Three consequences worth internalising now: + +1. **Metrics are cheap in traffic but expensive in cardinality.** Doubling your + request rate barely changes Prometheus's memory; adding `user_id` as a label + can kill it. Logs and traces are the opposite: cheap in dimensions, expensive + in volume. This asymmetry drives every design decision later. +2. **Traces are events with two extra fields:** a duration and a parent pointer. + That's the whole trick. Everything else (service maps, critical paths, span + metrics) is derived from those two fields. +3. In OpenTelemetry, "events" aren't a fourth top-level signal you export + separately. They show up as **span events** (a timestamped annotation attached + to a span — this is how `record_exception` works) and as **log records carrying + an `event.name` attribute**; the Events API is built on the Logs data model. + When Babak says M.E.L.T., he most likely means "logs plus the discrete + business/domain events you care about" — check with him, but treat it as + *structured logs and span events*, not a separate pipeline. + +### "Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?" + +**OpenTelemetry first, without much doubt.** + +Loki, Tempo, Prometheus and Mimir are *destinations*. They're databases with query +languages. OpenTelemetry is the **producer-side data model and wire protocol** — it's +the only part of this that touches your Python code, and it's the part that +determines what the destinations can possibly show you. If you learn Loki first you +learn Loki's label model and its promtail-isms, most of which don't transfer. If you +learn OTel first, every backend becomes "point OTLP at a different URL." + +One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly +early, in parallel with OTel metrics — because it *constrains what you're allowed to +emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them +(Phases 4–5), not up front. + +Filebeat and Logstash are a **different world entirely** (Elastic's). They are not +part of the Grafana stack and you do not need them for this task. Phase 9 covers +them as a judgment question, which is exactly how Babak framed them. + +### Verdict on the OneUptime article you found + +**Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a +teaching resource. Read it in **Phase 4/5**, not now; it assumes you already +understand spans, span metrics and PromQL. + +What's genuinely good in it, and rare to find in one place: + +- The `spanmetrics` and `servicegraph` **connectors** in the Collector — this is how + you get RED metrics and a service map "for free" from traces you already emit. + This is the single highest-leverage config in the whole article. +- The Tempo datasource provisioning YAML with `tracesToLogsV2`, `tracesToMetrics`, + `serviceMap` and `nodeGraph` — the click-through correlation that makes Grafana + feel like an APM. +- `exemplarTraceIdDestinations` on the Prometheus datasource — jump from a latency + spike on a graph straight into the trace that caused it. +- The tail-sampling policy block (keep all errors, keep all slow traces, sample the + rest). + +Two caveats: + +- Its Collector config exports logs via the **`loki` exporter**. Loki now accepts + **OTLP natively** on its own endpoint, and that is the current recommended path — + fewer moving parts, better attribute mapping. Use the native OTLP route. +- It's a vendor blog (OneUptime sells a competing product). The Grafana content is + accurate, but treat the official OpenTelemetry and Grafana docs as the source of + truth for anything spec-level. +- **Its `spanmetrics` / `servicegraph` Collector connectors do not apply to your + stack.** See the next section — your platform generates those metrics in Tempo + instead. The article's PromQL is still useful, but the metric names differ. + +--- + +## The platform Babak gave you (`observability-ansible`) + +This changes *where* your telemetry goes and *how the pipeline is configured*. It +does not change the learning arc, the app, or the test suite. Read this before +Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9. + +### What the repo is + +An Ansible project that deploys **two independent observability backends** on one +Ubuntu/Debian host via Docker Compose: + +1. **ELK + Grafana** — Filebeat → Logstash → Elasticsearch → Kibana/Grafana +2. **Grafana stack** — Prometheus + Alertmanager + Loki + Tempo + Grafana, with + **Grafana Alloy** as the collector and node-exporter for host metrics + +It contains **no application code and no dashboards**. It is the *platform half* of +Babak's task, handed to you pre-built. Your half is everything that produces +telemetry and everything that reads it. + +Deployment: `ansible-playbook playbooks/grafana-stack.yml` and +`playbooks/elk-grafana.yml`, idempotent, versions pinned in `group_vars/all.yml` +(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2, +Elastic 9.1.3). Everything lands under `/opt/observability/`. + +### The architecture + +Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns: +**Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline → +Explore & Correlate**, with three input classes feeding it. + +``` +App metrics ──/metrics, Prometheus exposition──┐ +Container logs ──Docker socket, tail──▶ Alloy ──push API──▶ Loki +Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo + ("spans + resource attrs") │ +Node Exporter :9100 ──────scrape──────┐ │ + ▼ │ + Prometheus ◀──remote_write────────┘ + (TSDB, PromQL, rules) span metrics + exemplars + │ + firing alerts ▼ + Alertmanager ──route──▶ Receivers + (group, dedupe) (Slack, Email, PagerDuty, + │ webhook, SMTP) + Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four + Correlation key: trace_id / exemplars +``` + +Note what the diagram says about the OTLP box: **"spans + resource attrs"**. Traces +only. And app metrics are labelled **"/metrics endpoint, Prometheus exposition"** — +not OTLP. The diagram and the code agree on this; it is a design decision, not an +oversight. Plan your instrumentation around it. + +### One place the diagram and the code disagree + +The diagram draws app metrics as `App metrics → scrape → Alloy → /metrics → +Prometheus`. The shipped config does not do that: + +- `config.alloy.j2` contains **no `prometheus.scrape` component at all** — Alloy + collects logs and receives traces, nothing else. +- `prometheus.yml.j2` scrapes `prometheus`, `node-exporter`, `loki`, `tempo`, + `alloy:12345`, `grafana`, then loops over `prometheus_extra_scrape_jobs`. So your + app targets are scraped by **Prometheus directly**, bypassing Alloy entirely. The + green `/metrics` arrow from Alloy to Prometheus in the diagram is Prometheus + scraping Alloy's **own** self-monitoring endpoint, not your app's metrics + passing through it. + +Neither is wrong; the diagram shows the intended architecture and the code +implements a simpler one. Closing that gap — adding `prometheus.scrape` + +`prometheus.remote_write` to Alloy so metrics really do flow through the collector — +is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth +mentioning to Babak.** Spotting that a diagram and its implementation have drifted is +exactly the skill this task is meant to build. + +### Four things in it that will cost you a day if you don't know them + +**1. Alloy's OTLP receiver only forwards traces.** The config is literally: + +``` +otelcol.receiver.otlp "default" { + grpc { endpoint = "0.0.0.0:4317" } + http { endpoint = "0.0.0.0:4318" } + output { + traces = [otelcol.exporter.otlp.tempo.input] + } +} +``` + +There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs +your app sends are accepted and then **silently discarded** — no error, no warning, +just nothing arriving in Grafana. This is the single most likely thing to eat an +evening. + +The diagram confirms this is deliberate: the OTLP input is labelled "spans + +resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition +input. So you have three options, and you should understand all three: + +- *Follow the platform's grain (simplest):* expose a Prometheus `/metrics` endpoint + per service and register them in `prometheus_extra_scrape_jobs` in + `group_vars/all.yml`. Prometheus scrapes your services directly. Logs continue to + arrive via stdout scraping. +- *Implement what the diagram draws:* add `prometheus.scrape` + + `prometheus.remote_write` components to Alloy so app metrics flow through the + collector, as the architecture intends. Buys you central relabeling and one place + to change collection policy. +- *Go all-OTLP:* add `metrics` and `logs` outputs to the receiver, plus + `otelcol.exporter.prometheus` → `prometheus.remote_write` and + `otelcol.exporter.loki` → `loki.write`. One protocol for all three signals — the + architecture you'd argue for on a greenfield platform, and the furthest from what + ships. + +Do the first to get moving, then do at least one of the others as a deliberate +exercise and write up the trade-off. That comparison is worth more than any single +answer. + +**2. Span metrics come from Tempo, not from the collector.** `tempo.yml.j2` enables: + +``` +metrics_generator: + storage: + remote_write: + - url: http://prometheus:9090/api/v1/write + send_exemplars: true +overrides: + defaults: + metrics_generator: + processors: [service-graphs, span-metrics, local-blocks] +``` + +So RED metrics, the service graph, and exemplars already exist — generated inside +Tempo and remote-written into Prometheus (which runs with +`--web.enable-remote-write-receiver`). **Do not add `spanmetrics`/`servicegraph` +connectors to Alloy** — you'd be generating the same series twice under different +names. The metric names differ from the OneUptime article's; go find the actual +names in Prometheus before writing any PromQL. (`local-blocks` also gives you +TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.) + +**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki +datasource is provisioned with: + +``` +derivedFields: + - datasourceUid: tempo + matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)' + name: TraceID +``` + +Your log lines must contain **lowercase** `trace_id` (or `traceid` / `trace-id`). +This matters more than it looks: OpenTelemetry's `opentelemetry-instrumentation-logging` +auto-injection emits **`otelTraceID`**, which does **not** match that pattern. +So on this platform, the structlog processor emitting a lowercase `trace_id` field +isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus +datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram states +this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything +in the Explore & Correlate column hangs off that one field name. + +**4. There is no dashboard provisioning at all.** The Grafana provisioning mount +contains only `datasources/`. Every dashboard is yours to add — which means Babak's +"create dashboards as code, destroy and recreate in seconds" requirement is entirely +your deliverable, and the natural home for it is *inside this Ansible role*. + +### What this changes, phase by phase + +| Phase | Impact | +|---|---| +| 0 / 0a | **None.** App and test suite are unaffected. | +| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. | +| 2 | Endpoint changes to Alloy; tracing work itself is identical. | +| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. | +| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. | +| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. | +| 6 | None. | +| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. | +| 8 | None, plus TraceQL metrics from `local-blocks`. | +| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. | +| 10 | Demo now includes `ansible-playbook` rebuilding the platform. | + +### Where to run it + +The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for +ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside +everything else will hurt. Use a separate VM (a cheap cloud instance or a local +Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing +at it. That also makes the deployment *real* rather than a localhost shortcut, which +is the point of it being Ansible. + +If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker +socket and will scrape **every** container's stdout, including Ledgerline's, +automatically — convenient, and worth knowing before you wonder where those logs +came from. + +### One thing to confirm with Babak + +Is this repo a **platform you must consume as-is**, or a **starting point you may +modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap +assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and +extend `group_vars`. If Babak intends it as a fixed platform, several +recommendations below change — ask before Phase 3. + +--- + +## Your resource spine + +Pick **one paid course as a backbone** and use the free material as targeted +supplements. Both Udemy options below ship captions, which matters for your +subtitle-to-notes workflow. + +**Backbone — choose one:** + +| Resource | Format | Length | Cost | Best for | +|---|---|---|---|---| +| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. | +| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. | + +If budget allows, buy both — they overlap by maybe 30% and the second half of each +is different. If only one: take the **OpenTelemetry** one, because instrumentation +is the part you have to *write*, and Grafana's own docs are excellent for the rest. + +**Free, and worth your time:** + +| Resource | Format | Length | Use in | +|---|---|---|---| +| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. | +| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. | +| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. | +| **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. | +| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. | +| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. | +| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 | +| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. | +| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. | +| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on | +| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 | +| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. | +| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. | +| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 | +| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. | +| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 | +| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 | + +**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on +your k8s cluster until everything works locally — debugging instrumentation and +debugging Kubernetes at the same time is how these projects die. Porting it to the +cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half +Babak has planned for you. + diff --git a/docs/roadmap/phase-0.md b/docs/roadmap/phase-0.md new file mode 100644 index 0000000..8ff9238 --- /dev/null +++ b/docs/roadmap/phase-0.md @@ -0,0 +1,127 @@ +# Phase 0 — Build the base project, test-first + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`intro.md`](intro.md) · Next: [`phase-1.md`](phase-1.md)* + +**Status:** Done + +**Message:** Both halves complete and verified against the live stack, not just +collected. + +- **0a (test suite):** 81 tests across `tests/` (11 files + a 5-module + harness), covering all eleven goals G1–G11 and all 13 named pathologies. + Verified genuinely red against the empty repo before any implementation + existed; `tests/RED-BASELINE.md` records that run. `tests/README.md` + documents the invented request-contract fields, measurement methodology, + and known-flaky tests. +- **0b (implementation):** all seven services (`gateway`, `ledger`, `fraud`, + `sanctions`, `insights`, `notifier`, `audit`), `db/schema.sql` + + `optional_indexes.sql`, `scripts/seed.py` + `smoke.py`, the frontend, and + `docker-compose.yml`. Seeded: 800 accounts, 250,000 transactions, 45 + merchant categories, 12,000 sanctions entries, in ~12s. `make smoke` and + `make test-contract` (64/64) pass; `make test-baseline` and + `make test-pathology` pass except for a handful of tests documented below + as load-sensitive rather than broken. +- **Two changes to the frozen test contract, both made only after flagging + them and getting an explicit go-ahead — not applied unilaterally:** + - `test_row_counts_match_manifest_and_targets` loosened the `transactions` + row-count check from exact equality to `>=`, since other contract tests + in the same `make test-contract` run legitimately insert rows first. + - `test_pool_starvation_plateaus_throughput` was **removed outright**. + Its baseline assertion (throughput scales ≥2.5x from concurrency 5→25) + is incompatible with this same spec's own §3 mandate of one uvicorn + worker per service — confirmed by reproducing the identical + non-scaling behaviour on a bare `GET /healthz` with zero backend + logic. `pool_starvation` itself is still fully implemented + (`services/ledger/db.py`'s dual-engine pool selection); only that one + differential test was dropped. +- **Known-flaky, documented rather than papered over:** + `test_clean_baseline_is_healthy`, `test_nightmare_degrades_but_survives`, + the baseline half of `test_blocking_event_loop_serializes_concurrent_requests`, + and `test_missing_index_forces_seq_scan` compare a fast-path measurement + against a slow-path one under real concurrent load, and pass at roughly a + 60–80% rate on ordinary dev hardware depending on momentary host + contention. See "Known-flaky tests" in `tests/README.md`. +- **Notable implementation deviations** (none change observable behaviour of + the 13 pathologies or the request/response contract): gateway's `GET /` + serves the frontend rather than §4's generic info JSON (the other five + HTTP services still return that JSON as specified); `insights` splits a + cheap PK-only `quick_summary` (embedded in every transaction response) + from the real cached/N+1-affected 120-day analytics endpoint, so the + missing-index pathology doesn't tax every request; `retry_storm` fires + all 4 attempts unconditionally rather than stopping at first success + (spec's "no circuit breaker" taken literally — also the only way to reach + the required lookup-multiplication ratio given `sanctions_errors`' fixed + 3% rate); `poison_data` resolves via a Redis hash populated at seed time + instead of a live table scan, to avoid the deliberately missing index; + uvicorn's access log is disabled fleet-wide (redundant with the spec's + own logging design, and was double-counting two log-line-based pathology + tests); Postgres `max_connections` raised from 100 to 300 (five + services' pool sizes can sum past the default under load). + +**Time-box:** 2–3 evenings. Two agent sessions plus one evening of your own +reading. + +**Deliverable:** the working Ledgerline system with no observability, plus a +black-box integration suite that pins down both its working state and its +deliberate brokenness. + +### Why TDD here, when the app itself doesn't matter + +The microservices are throwaway. The *properties* aren't. Everything from Phase 2 +onward assumes the system still behaves a specific way: seven real services in +the path, async work genuinely off the request path, five hot accounts owning a +third of the rows, and thirteen pathologies that actually degrade something. You +will be editing all seven services for weeks. Without a suite that fails when one +of those properties quietly breaks, you'll spend an evening chasing a trace +anomaly that turns out to be a regression you introduced yesterday. + +Writing the tests first also forces the contract to be explicit before anything +is built — which is what makes a second agent able to implement against it +without guessing. + +### 0a — the test suite + +Agent session one, from `phase-0a-test-suite-spec.md`. That agent writes only +`tests/` and must not create so much as a stub under `services/`. Its output +should be **red**, with a `tests/RED-BASELINE.md` recording why each test fails +right now — a test that passes against an empty repo is a broken test. + +The suite asserts goals, not mechanics. Two representative examples worth +understanding before you read the code: + +- *"Stop the `sanctions` container and transactions must fail."* This is the test + that catches an implementation quietly collapsing seven services into one + process. Nothing else would. +- *"With `n_plus_one` on, one insights request issues ≥ 50 SQL statements."* + Measured via `pg_stat_statements`, not wall clock — because a query count + can't be faked with a `sleep()`, and a latency threshold can. + +That second point is the design principle throughout: **prefer assertions that +are hard to satisfy without the real mechanism.** Query counts, `EXPLAIN` plan +node types, Redis consumer-group lag, container RSS, and genuine unhandled +exception paths all have that property. + +### 0b — the implementation + +Agent session two, from `phase-0-base-project-spec.md`, with `tests/` handed over +as a **read-only contract**. That agent may not edit, skip, `xfail` or loosen a +single test. If something looks impossible it stops and reports rather than +adjusting the target. + +Done when `make test` is green three consecutive runs and +`git diff --stat tests/` is empty. + +### Then, yourself + +1. **Read every service's `app.py` end to end.** You need to know this codebase + cold, because from Phase 2 onward you're modifying all seven services. +2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and + then try to figure out what's wrong using only `docker compose logs`.** Give it + twenty real minutes. You will fail, and the specific way you fail — no timing, + no causality, no cross-service correlation, a `X-Request-Id` that dies at the + Redis Stream — is the motivation for everything that follows. Write down what + you wished you had. That list is your acceptance criteria for Phase 8. + +**Done when:** `make test` is green three runs in a row, all nine acceptance +criteria in the base spec pass, and you have that written list of frustrations. + diff --git a/docs/roadmap/phase-1.md b/docs/roadmap/phase-1.md new file mode 100644 index 0000000..e5ff039 --- /dev/null +++ b/docs/roadmap/phase-1.md @@ -0,0 +1,89 @@ +# Phase 1 — The model, and one signal end to end + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)* + +**Status:** Not started + +**Message:** Blocked on nothing — Phase 0 is done and the platform is +described above. Waiting to begin. + +**Time-box:** one evening. + +**Why:** you need the vocabulary, but more importantly you need to see one trace +appear in Grafana within an hour, so the rest of the roadmap feels concrete rather +than abstract. + +### Concepts + +- **Monitoring vs observability.** Monitoring answers questions you knew to ask + (known unknowns): dashboards and alerts built from predefined aggregates. + Observability is a property of the system: can you ask a *new* question about + production, that you never anticipated, without shipping new code? Charity Majors' + framing — monitoring is known-unknowns, observability is unknown-unknowns. The + practical implication for you: emit **wide, high-dimensional events** (rich span + attributes), not just pre-aggregated counters. +- **The OTel architecture split: API vs SDK vs Collector.** + - The **API** is what your code and third-party instrumentation libraries call. + If no SDK is configured, API calls are no-ops — that's deliberate, it means a + library can be instrumented without forcing a dependency on any implementation. + - The **SDK** is the engine: sampling, batching, context propagation, export. + Application owners configure the SDK; library authors only touch the API. + - The **Collector** is a standalone process that receives, processes and exports + telemetry. Optional, but you should use one. +- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318. +- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`, + `service.version`, `deployment.environment`) and is attached to every signal from + that process. Attributes describe the individual span/metric/log. +- **Semantic conventions**: standardised attribute names (`http.request.method`, + `db.system`, `server.address`). Use them for anything standard; namespace your own + (`ledgerline.account.id`). This is what makes cross-service, cross-language + correlation work — and it's what Grafana's prebuilt dashboards key off. +- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha — + aware of, not learning yet.) +- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your + app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of + seven services. The Collector also does the things you don't want in your app: + retry, PII redaction, tail sampling, memory limiting, fan-out to multiple + destinations. + +### Resources + +- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the + M.E.L.T. and context-propagation sections; skip the Node code. +- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min. +- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop + repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand + it — just see what "done" looks like. + +### Build + +1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it, + `ansible-galaxy collection install -r requirements.yml`, change + `grafana_admin_password` (vault it), then + `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the + README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093, + Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318. + Then open the Alloy UI and **read the component graph** — it shows you the exact + pipeline, including the fact that the OTLP receiver has only a traces output. + Seeing that in the UI before you get bitten by it is worth ten minutes. + *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`, + a single container with everything pre-wired. Useful for a first trace, but + don't build on it — its pipeline differs from your real platform in exactly the + ways that matter.)* +2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code + auto-instrumentation: + ``` + pip install opentelemetry-distro opentelemetry-exporter-otlp + opentelemetry-bootstrap -a install + ``` + and change its command to + `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003` + with `OTEL_SERVICE_NAME=sanctions`, + `OTEL_EXPORTER_OTLP_ENDPOINT=http://:4318`, + `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at + Tempo directly. +3. Fire a request. Open Grafana → Explore → Tempo → find the trace. + +**Done when:** you can see a single-service trace in Tempo, and you can explain out +loud what the Resource attributes on it mean and where they came from. + diff --git a/docs/roadmap/phase-10.md b/docs/roadmap/phase-10.md new file mode 100644 index 0000000..22a0f23 --- /dev/null +++ b/docs/roadmap/phase-10.md @@ -0,0 +1,40 @@ +# Phase 10 — Package the delivery + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)* + +**Status:** Not started + +**Message:** Depends on every prior phase being done — this is the wrap-up, +not independent work. + +**Time-box:** one evening. + +Babak gave you a task; deliver it like an engineer, not like a student. + +1. **`README.md`** — architecture diagram, one-command start, what each container is + for, and a "here's the demo" script. +2. **The five-minute demo path**, rehearsed: + - Delete every dashboard in the Grafana UI, then `ansible-playbook + playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt + from code, one command. + - Submit one transaction from the frontend, then follow it through Grafana: RED + dashboard → latency spike → exemplar → trace → span → logs for that span → back + to the trace. One request, all three signals, no searching. + - `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire, + and the service map turn red at a specific node. + - Show `docs/INVESTIGATION.md` and how you found the pathologies. +3. **The written answers** Babak asked for, as documents in the repo, not as a + conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`, + `docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99, + RTO/RPO/SLA/SLO/error budget in your own words. +4. **The test suite as part of the story.** Show `make test` green, then show the + three generations of the call-counting assertion side by side — + `docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what + each signal bought you. Most people demo dashboards; almost nobody can show + their observability work paying off inside their own test suite. +5. **A "what I'd do differently in production" section.** Tail sampling instead of + 100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet; + log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what + you *didn't* do, and why, is the strongest signal of seniority in the whole + delivery. + diff --git a/docs/roadmap/phase-2.md b/docs/roadmap/phase-2.md new file mode 100644 index 0000000..9f8451c --- /dev/null +++ b/docs/roadmap/phase-2.md @@ -0,0 +1,116 @@ +# Phase 2 — Distributed tracing + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)* + +**Status:** Not started + +**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first. + +**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task. + +**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap +(Resource, providers, exporters, Collector wiring) is shared across all three +signals — do it once here and logs and metrics become incremental. Second, tracing +is what Babak actually asked for ("trace a single request, see where it stuck"), +and once trace IDs exist, log correlation in Phase 3 works on the first try instead +of needing rework. + +### Concepts + +- **Trace / span / span context.** A trace is one request's journey; a span is one + timed operation within it. Each span carries a trace ID, its own span ID, and its + parent's span ID. That parent pointer is the entire data structure. +- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an + outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting + these right is what makes the service graph render correctly. +- **Context propagation.** + - *In-process*: OTel uses Python `contextvars`. This works across `await` — but + **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you + carry the context explicitly. + - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and + `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.) + - The classic bug: any boundary that doesn't carry those headers **starts a brand + new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with + `TraceContextTextMapPropagator().inject(...)` into the message payload on the + producer and `.extract(...)` on the consumer, then starting the consumer span + with that context as parent. This is the single most valuable exercise in the + phase — most people never do it manually and never really understand it. + - Related: `links` (for a consumer span that batches messages from many traces) + and **Baggage** (arbitrary key/values propagated alongside trace context — + useful, and a foot-gun, because it's sent on every hop and is visible to + downstream systems; never put secrets in it). +- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP + client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual + gets you the business semantics: `score_transaction`, `check_sanctions_list`, + `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces + that show *which service* was slow but not *why*. +- **Span attributes vs metric labels.** Spans are the right home for + high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale + with volume, not with distinct values. This is why "put it on the span, not the + metric" is the standing rule. +- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`. + That's how the exception ends up attached to the trace and how the `spanmetrics` + connector in Phase 4 will count your error rate. +- **Sampling — the concept now, the config later.** + - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops + errors at the same rate as successes. + - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow + traces, but must run in the Collector and buffer spans in memory. + - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off + now; you'll configure tail sampling in the Collector in Phase 5. + - Whatever you choose, the sampling decision must **propagate** — the sampled flag + rides in `traceparent`. Use `ParentBased` samplers so a service doesn't + unilaterally drop half of a trace its caller decided to keep. + +### Resources + +- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30 + min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both. +- Your chosen Udemy backbone: the instrumentation + context-propagation sections. +- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most + practical FastAPI-specific setup, including batch/queue env vars. +- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not + reading. +- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services + side by side. Look specifically at how they mix auto-instrumentation with + hand-rolled business spans, and how the Kafka producer/consumer propagate context. + This is the "professional implementation" you asked to study. + +### Build + +1. **Shared bootstrap.** Create `common/telemetry.py` with a single + `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`, + `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every + service calls it once at startup. Keep it under 60 lines. +2. **Auto-instrument all seven services**: `FastAPIInstrumentor`, + `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify + you now get a connected trace across `gateway → ledger → fraud → sanctions`. +3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`, + `sanctions.lookup_watchlist`, `insights.aggregate_120d`, + `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and + namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`, + `ledgerline.tx.amount_minor`). +4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span, + inject trace context into the message fields. Consumers (`notifier`, `audit`): + extract, and start a `CONSUMER` span with the extracted context as parent. +5. **Record exceptions** everywhere you currently `logger.exception(...)`. +6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly + more. +7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded + the "before" state so you could measure against it, and from here it's + actively wrong. Everything else in `tests/` must still pass: instrumenting + the system is not allowed to change its behaviour, and if `make test` goes red + after adding spans, you've broken something real. That is exactly the safety + net you built it for. + +### Done when + +- One request produces **one connected trace** covering all seven services, + including both async consumers. +- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that + ate the time. +- You can explain why `blocking_event_loop` looks different in a trace than + `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap* + before spans start, plus every concurrent request in that process degrading + together.) + diff --git a/docs/roadmap/phase-3.md b/docs/roadmap/phase-3.md new file mode 100644 index 0000000..98291bd --- /dev/null +++ b/docs/roadmap/phase-3.md @@ -0,0 +1,99 @@ +# Phase 3 — Structured logs, correlated to traces + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)* + +**Status:** Not started + +**Message:** Depends on Phase 2 (trace IDs must exist before log correlation +means anything). + +**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win. + +### Concepts + +- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with + events: a stable event name plus typed fields (`event="transaction_settled"`, + `account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names. + Consistent field names across all seven services — `account_id` everywhere, never + `accountId` in one place and `acct` in another. This is what makes LogQL queries + possible at all. +- **Trace–log correlation.** Every log line emitted inside an active span must carry + `trace_id` and `span_id`. This is what turns "I have logs and I have traces" into + "one pane of glass": click a span → see its logs; click a log line → open its + trace. Both directions are configured, and you'll set both up. +- **Two ways to ship logs, and which one your platform actually does:** + - **stdout → collector scraping → Loki.** The container-native way. Your app just + prints JSON; the pipeline handles the rest. Decoupled, survives app crashes, + standard in Kubernetes. **This is the only path wired in your Alloy config** + (`loki.source.docker` reading the Docker socket). + - **OTLP log exporter → collector → Loki.** Logs travel the same path as traces, + with the same Resource attributes attached automatically. Cleaner correlation, + but the app owns delivery. **On your platform this currently goes nowhere** — + Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and + dropped in silence. + - Do both anyway: use stdout for most services, and *extend Alloy* to wire the + OTLP logs path for one or two. Then argue the trade-off from experience. That + comparison — including the fact that a misconfigured pipeline fails silently + rather than loudly — is precisely what Babak will probe. +- **The field name is not a style choice here.** Your Loki datasource is provisioned + with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive. + A log line must carry lowercase `trace_id`. OpenTelemetry's logging + auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get + logs and traces both arriving, no errors anywhere, and no clickable link. Hence + structlog with an explicit processor rather than the auto-injection shortcut. +- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce + your entire fleet, and Loki bills/indexes on volume. +- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive + `json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build + redaction into the processor chain, not into each call site. +- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*, + not log content. Labels should be few and low-cardinality (`service`, `env`, + `level`). Everything else lives in the log line body and is filtered at query time. + Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it + in the JSON body and use a **derived field** to make it clickable. + +### Resources + +- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and + Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor + setup, including the hex formatting for trace/span IDs. +- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for + `OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID` + fields, if you'd rather not hand-roll a processor. +- Udemy backbone: the Loki + LogQL sections. +- Grafana docs → Loki → *Send data via OTLP* and *derived fields*. + +### Build + +1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a + processor that pulls the current span context and injects `trace_id` / `span_id`. +2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per + handler: request received (or rely on the span), business decision, error. +3. Add a redaction processor with a denylist of field names. +4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki + with a `container` label (Alloy's relabel rule sets it from the Docker container + name). Verify the `trace_id` field is present *and lowercase* in the JSON body. +5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service, + send some records, and watch nothing appear in Loki. Then check the Alloy UI: + the receiver shows traffic, and the pipeline shows no logs output. Now extend + `config.alloy.j2` — add a `logs` output on the receiver plus an + `otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch + them arrive. Write down what you'd have done to debug this if you *hadn't* known. + That is the most transferable half-hour in the whole project. +6. Confirm the provisioned correlation works in both directions: Loki's + `derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2` + (`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you + are verifying, not writing. If a link doesn't appear, the regex didn't match + your field name — check that before anything else. +6. **Migrate the test suite's log-counting helpers.** Three pathology tests + (`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines + in `docker compose logs` — the only cross-service call counter that existed at + Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki + with LogQL over structured fields instead. This is a small change and a + pointed one: notice how much shorter and more precise the assertion gets, and + that it now works without shelling out to Docker at all. That delta is the + value of structured logging, measured on your own code. + +**Done when:** from a slow trace you can click into that request's logs, and from +any log line you can click into its trace — in both directions, in two clicks. + diff --git a/docs/roadmap/phase-4.md b/docs/roadmap/phase-4.md new file mode 100644 index 0000000..4d884b7 --- /dev/null +++ b/docs/roadmap/phase-4.md @@ -0,0 +1,126 @@ +# Phase 4 — Metrics + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)* + +**Status:** Not started + +**Message:** Not blocked technically (metrics don't strictly need tracing +first), but sequenced after Phase 3 per the roadmap's ordering. + +**Time-box:** 2 evenings. + +### Concepts + +- **Instrument types.** + - **Counter** — monotonic, only increases. Requests, errors, messages consumed. + - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests, + connection-pool checkouts. + - **Histogram** — a distribution. Request latency, payload size. This is what + gives you percentiles. + - Synchronous instruments are recorded inline; **observable/asynchronous** ones are + read via a callback on collection (right for things you *poll*, like pool size). + - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore + **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic + nonsense. Use histograms. +- **Cardinality — the one thing that will bite you.** Every unique combination of + metric name + label values is a separate time series in memory. Add `account_id` + as a label on a 800-account demo and you've made 800 series; do it in production + and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed + ~10 000 distinct values in production, it does not belong on a metric.** Put it on + a span. Histograms multiply this (one series per bucket per label combination), so + don't over-bucket either. + - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not + the concrete path), `http.status_code` class, `kind`, `currency`. + - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`, + raw URL, timestamp, error message. +- **RED and USE.** + - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your + per-service dashboard. + - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory, + connection pools, queues, disk. Saturation is the one everyone forgets and the + one that predicts collapse. + - You want both: RED tells you the service is slow, USE tells you which resource + is why. +- **Span metrics — generate metrics from traces.** Two architectures exist for this, + and knowing both is the point: + - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and + `servicegraph` connectors read spans in flight and emit RED metrics and edge + metrics. This is what most blog posts (including the OneUptime article) show. + - *In the trace store:* Tempo's **metrics-generator** does the same job after + ingestion, remote-writing the result to Prometheus. **This is what your platform + does** — `processors: [service-graphs, span-metrics, local-blocks]` with + `send_exemplars: true`, remote-writing to a Prometheus started with + `--web.enable-remote-write-receiver`. + - So RED metrics, the service map and exemplars **already exist** before you write + a single metric line. Do not add the collector connectors as well — you'd + duplicate every series under a second set of names. **Go find the real metric + names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL; + they will not match the article's. + - The trade-off worth being able to argue: collector-side generation is cheaper + and works before storage, but sees only what passes through that collector; + Tempo-side generation sees everything that landed, survives collector restarts, + and can be enabled per-tenant — at the cost of load on the trace store. +- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you + click a spike on a latency graph and land in the exact trace. On your platform + this is already wired end to end: Tempo's generator sets `send_exemplars: true`, + and the Prometheus datasource declares `exemplarTraceIdDestinations` with + `name: trace_id`. If you emit your own histograms with exemplars, the label must + be `trace_id` to match. +- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you + can do either — receive OTLP from apps, then either expose a `/metrics` endpoint + for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that + the pull model is why Prometheus has service discovery and why `up` exists. + +### Resources + +- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min). + Read the first half. Best cardinality explanation available. +- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill, + not a fact. +- **The OneUptime APM article** — now it's time. Take its `spanmetrics` / + `servicegraph` connector config and its RED PromQL queries almost verbatim. +- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics. + +### Build + +1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter. +2. Custom application metrics, deliberately few: + - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`) + - `ledgerline.transaction.duration` (Histogram; labels: `kind`) + - `ledgerline.fraud.score` (Histogram; no labels) + - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`) + - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`) +3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP + metrics, so pick a path and implement it: + - *Platform grain:* expose `/metrics` on each service and add them to + `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the + playbook. Prometheus scrapes your services directly. Note the irony and the + lesson — you're back to a pull model and a second client library, which is + exactly the fragmentation OTel exists to fix. + - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to + Alloy so app metrics flow through the collector as the architecture drawing + intends. This also closes a real diagram-vs-code gap in the repo. + - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire + `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for + all three signals. + Do the first to get moving; do one of the others afterwards and write up which + you'd choose for a real platform team and why. That write-up is a Phase 9 + artefact. +4. Deliberately break it: add `account_id` as a label to + `ledgerline.transactions.total`, run a 200-request burst, and watch the series + count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now + you'll never forget. +5. Compare your hand-written duration histogram against the `spanmetrics`-generated + one. Understand why they differ slightly and when you'd want each. +6. **Second migration of the call-counting assertions.** The three tests you moved + to LogQL in Phase 3 can now count **client spans** in Tempo instead — a + TraceQL query for `sanctions` client spans within a single trace answers + "how many lookups did this one transaction make?" directly, which is the + question the test was always really asking. Do it and compare all three + generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the + clearest possible demonstration of what each signal is actually for, on a + problem you already understand. + +**Done when:** you have a RED view per service, a working service map, and you can +click a latency spike through to a trace via an exemplar. + diff --git a/docs/roadmap/phase-5.md b/docs/roadmap/phase-5.md new file mode 100644 index 0000000..9bbfc1b --- /dev/null +++ b/docs/roadmap/phase-5.md @@ -0,0 +1,76 @@ +# Phase 5 — Dashboards and alerts as code + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)* + +**Status:** Not started + +**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth +dashboarding. + +**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and +recreate in seconds.* + +### Concepts + +- Three levels of "as code", and you should know when each applies: + 1. **File provisioning** — YAML in `provisioning/datasources/` and + `provisioning/dashboards/`, plus dashboard JSON files mounted into the + container. Grafana loads them at startup. Simplest, no state, no API tokens, + perfect for a Compose demo. **This alone satisfies the requirement.** + 2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java, + PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a + function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable + diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead + of Grafonnet (which is legacy and unofficial). + 3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed, + multi-environment, or GitOps setups. Overkill here; the right answer *later*, + when this lands on your Kubernetes cluster. +- **Dashboards should answer questions, not display data.** A good service dashboard + answers, in order: is it up, is it serving, is it erroring, is it slow, and where + is the pressure. Panels that nobody looks at during an incident are cost, not value. +- **Alerts as code too.** Grafana alert rules, contact points and notification + policies all provision from YAML. And keep the *alert definition* next to the + dashboard that explains it. + +### Resources + +- Grafana docs → *Observability as code* (the index page — it lays out Foundation + SDK, file provisioning, Git Sync and Terraform, and tells you which to use when). +- Grafana Foundation SDK repo → the Python examples, especially the `red-method` + example. +- Udemy backbone: dashboard design sections (panel types, transformations, + variables, dashboard best practice). + +### Build + +Your platform provisions **datasources only** — there is no dashboards provider at +all. So this phase is entirely yours, and the right home for it is inside the +Ansible role, using the same mechanism as everything else. + +1. **Extend the role.** Add + `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2` + (a file-based dashboard provider pointing at a mounted directory), a + `config/grafana/dashboards/` directory for the JSON, and the tasks to template + and copy them. The Compose template already mounts + `./config/grafana/provisioning` read-only, so the provider is picked up on + restart. +2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana + Foundation SDK (Python)**: one function producing a RED dashboard for a given + service name, looped over the seven services; a second producing a system + overview (service graph, stream lag, pool saturation, error-budget burn). Commit + the generator *and* its output; the Ansible role ships the output. +3. **The demo is now `ansible-playbook`.** Delete every dashboard through the + Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back. + That is a strictly stronger answer to Babak's "destroy and recreate in seconds" + than a local Makefile target, because it's the same command that builds the + platform. +4. While you're in there, review the provisioned datasources and add what's missing: + `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a + worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping — + understand why `filterByTraceID: true` alone is sufficient here, and when it + wouldn't be. +5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice. + +**Done when:** re-running the playbook restores every dashboard from scratch, and no +dashboard has ever been saved through the Grafana UI. + diff --git a/docs/roadmap/phase-6.md b/docs/roadmap/phase-6.md new file mode 100644 index 0000000..8bf66f3 --- /dev/null +++ b/docs/roadmap/phase-6.md @@ -0,0 +1,94 @@ +# Phase 6 — Load testing with k6, and percentiles + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)* + +**Status:** Not started + +**Message:** Depends on Phase 4 (metrics to plot k6's output against) and +benefits from Phase 5's dashboards existing. + +**Time-box:** 1–2 evenings. + +### Concepts — P95 and P99 properly + +You asked about these specifically, and there's more to them than "the 95th +percentile." + +- **A percentile is meaningless without a population and a window.** "p95 = 400 ms" + is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms" + is. +- **You cannot average percentiles.** Not across instances, not across time buckets. + `avg(p99)` is wrong and produces a number that looks plausible. The correct move is + to aggregate the *histogram buckets* and compute the quantile from the sum: + `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`. +- **Prometheus classic histograms give you an interpolated estimate**, bounded by + your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that + bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that + bracket your actual SLO threshold. (Native histograms largely solve this and are + worth knowing about.) +- **Which population?** An overall p99 across all endpoints is dominated by whichever + endpoint has the most traffic. Slice by route, or you'll miss that + `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine. +- **Tail latency amplification** — the most important idea here for a microservice + system. If one user request fans out to 10 backend calls, each with an independent + p99 of 1 s, then the chance the *user request* touches at least one slow call is + `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's + p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is + fine" and "the product is slow" coexist. Ledgerline's fan-out makes this + measurable — go measure it. +- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable, + most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly + that. +- **Coordinated omission** — a load-generator trap. If your generator waits for a + response before sending the next request, then when the system stalls, the + generator stalls too and simply *doesn't record* the requests that would have been + slow. Your measured latency is systematically optimistic. Fix: use an **open + model** — a fixed *arrival rate* independent of response times. In k6 that means + the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than + `constant-vus`. Knowing this is a genuine senior-level differentiator. + +### Concepts — k6 + +- **Test types:** smoke (does it work at all), average-load (normal traffic), + stress (past normal, find the knee), spike (sudden surge, test recovery), + soak (hours, find leaks and lag drift), breakpoint (ramp until failure). + Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak. +- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model + (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above. +- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't + fail the run; **thresholds are SLOs expressed in code and they set the process exit + code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`. +- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's + view and the application's view sit on the same Grafana time axis. That side-by-side + is the single most convincing screenshot you can show Babak. +- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're + writing 60 lines of it. + +### Resources + +- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*, + *Thresholds*. Free, genuinely well written; ~2 h to work through. +- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free, + hands-on). +- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one. +- Grafana docs → *k6 output to Prometheus remote write*. + +### Build + +1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`, + sharing a `common.js` for setup and payload generation against random accounts. +2. Use **arrival-rate executors** for `average` and `stress`. Write down why. +3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`, + `http_req_failed rate<0.01`. +4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard. +5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result + sets. This before/after pair is your baseline for Phase 8. +6. Optional but impressive: have k6 send W3C trace context headers so load-test + requests appear as traces, letting you click from a k6 spike into a specific + trace. Check the current k6 docs for the tracing module's status — it has moved + between experimental and stable. + +**Done when:** `make load-stress` produces a Grafana view where you can see the +arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the +service that broke first. + diff --git a/docs/roadmap/phase-7.md b/docs/roadmap/phase-7.md new file mode 100644 index 0000000..cab7cc8 --- /dev/null +++ b/docs/roadmap/phase-7.md @@ -0,0 +1,91 @@ +# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)* + +**Status:** Not started + +**Message:** Depends on Phase 6's load-test baseline (the latency SLO should +come from measured numbers, not be picked from thin air). + +**Time-box:** 1–2 evenings. Mostly thinking, a little config. + +### Concepts + +- **SLI** — a *measurement* of something users experience, expressed as a ratio of + good events to valid events. Not "CPU is at 60%". Rather: *the proportion of + `POST /api/transactions` requests that returned 2xx within 500 ms.* +- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction + requests succeed within 500 ms, measured over a rolling 28 days." +- **SLA** — an external, *contractual* promise, with consequences (refunds, credits). + Always **looser** than your SLO, so you have room to miss internally without + breaching commercially. If your SLA and SLO are the same number, you've made a + mistake. +- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1% + budget; over 30 days that's about **43 minutes**. The budget is the thing that + turns reliability from an argument into arithmetic: if there's budget left, ship + features and take risks; if it's spent, the policy says you stop feature work and + spend on reliability. That policy — agreed in advance by dev, ops and product — is + the whole point. +- **Burn rate.** How fast you're consuming budget relative to the window. + Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's + multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket) + is what stops you from either paging on every blip or noticing the slow bleed a + month late. +- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs, + and it's worth being precise because interviewers probe exactly here. + - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long + from failure to restored service. It drives your architecture: replication, + warm standby, failover automation, runbook quality. + - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured + in time: how far back can you afford to lose. It drives your **backup and + replication frequency**. RPO of 15 minutes means backups/WAL shipping at least + every 15 minutes. + - The relationship worth stating out loud: **SLO governs steady-state quality; RTO + and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says + "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most + 5 minutes of transactions." Both feed the SLA. + - Both are **business decisions, not technical ones** — you derive them by asking + what an hour of downtime or an hour of lost data actually costs, then buying the + cheapest architecture that meets it. Tightening either is expensive and + non-linear. + +### Resources + +- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is + immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free. +- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks, + all free PDFs. Run the exercises solo with Ledgerline as the subject. +- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable + free, self-paced, deeper. +- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough + overview; pair it with your Postgres backup knowledge. + +### Build + +1. Write `docs/SLO.md`. For Ledgerline define, with justification: + - Availability SLI/SLO on `POST /api/transactions` + - Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not + from thin air) + - A freshness/lag SLI on the notifier consumer group + - A stated SLA that is deliberately looser than each SLO + - An RTO and RPO for the Postgres instance, with the backup cadence they imply + - An error budget policy: what happens at 50%, 75% and 100% consumed +2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**. + Your platform ships **Alertmanager** plus a Prometheus rules directory and the + sample rules `HostExporterDown` and `HostHighCPU`, so the grain is + *Prometheus rule files → Alertmanager*, templated by the Ansible role, with + `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but + be able to argue the alternative: Grafana unified alerting can evaluate across + datasources (including Loki queries) and lives with your dashboards, whereas + Alertmanager is metrics-only but is the ecosystem standard and gives you + grouping, deduplication, inhibition and silences for free. Note which you chose + and why in `docs/SLO.md`. +3. Add an error-budget panel to the overview dashboard: budget remaining, current + burn rate, projected exhaustion. +4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and + watch the fast-burn alert fire. + +**Done when:** you can explain, in one paragraph each and without notes, the +difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule +decision. + diff --git a/docs/roadmap/phase-8.md b/docs/roadmap/phase-8.md new file mode 100644 index 0000000..fd5e85e --- /dev/null +++ b/docs/roadmap/phase-8.md @@ -0,0 +1,90 @@ +# Phase 8 — Bottleneck hunting (the capstone) + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)* + +**Status:** Not started + +**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics, +dashboards, load-test baselines, and burn-rate alerts are the toolkit this +phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts. + +**Time-box:** 2 evenings. This is where the whole thing pays off. + +### Concepts — how to actually answer "how do I find bottlenecks?" + +Babak asked how you'd *think* about this, not for a list of tools. Here's the +method, roughly in order. The discipline is to go **top-down from the user, not +bottom-up from the infrastructure** — bottom-up is how people spend three hours +tuning a database that wasn't the problem. + +1. **Start at the SLI, not the server.** Which endpoint, which percentile, which + window, compared to what baseline? "It's slow" is not a starting point. "p95 of + `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is. +2. **Localise to a service** using RED + the service graph. Which node's duration + grew? Note that a slow *downstream* makes its *upstream* look slow too, so read + the graph from the leaves inward. +3. **Get to exemplar traces.** Click the latency spike through to a trace, or query + TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want + several slow traces plus one fast trace of the same endpoint as a control. +4. **Read the trace for five specific shapes:** + - **A long span** — the work itself is slow. Look at its attributes: which query, + which host, which parameters. + - **A gap** — wall-clock time not covered by any span. This is the interesting + one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a + blocked event loop, or a missing instrumentation library. Ledgerline's + `pool_starvation` and `blocking_event_loop` both show up as gaps. + - **Serialisation** — spans laid end to end that could have been concurrent. + Ledgerline's `no_batching` looks exactly like this. + - **Repetition** — the same span name 150 times in a row. That's your N+1. + - **Retries** — the same client span repeated 3–4 times with errors. That's + `retry_storm`, and note that it makes an *overload* worse, never better. +5. **Separate latency from saturation.** Switch from RED to USE on the suspected + resource: connection pool in-use vs pool size, stream consumer lag, CPU, + memory. A resource at 100% utilisation with a growing queue is saturated; + latency there is a *symptom*, not the cause. +6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency + equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10 + concurrent slots. If your pool is 5, you queue — and queueing latency grows + non-linearly as you approach saturation. This one formula explains most + connection-pool and worker-count incidents you'll ever see. +7. **Check for tail amplification** (Phase 6) before blaming any single service. +8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one + chaos flag, re-run the *identical* k6 profile, and compare the *same percentile* + over the *same duration*. One variable at a time. Never declare a fix from a + dashboard that "looks better." +9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest + becomes the bottleneck. Re-measure after every change; stop when you're inside + the SLO, not when you run out of ideas. +10. **Know what your tools can't see.** Traces won't show you CPU time inside a span, + or lock waits inside Postgres, or which line of Python burned the cycles. That's + where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous + profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over. + +### Resources + +- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing + for saturation. +- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk). +- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best + thing ever written on why microservice latency behaves the way it does. +- Grafana docs → Tempo → **TraceQL**. You'll live in this. +- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know + these; connect them to the trace's DB spans. + +### Build + +1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at + `docs/PATHOLOGIES.md`.** +2. Run `make load-average` for 10 minutes. +3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven, + like a real incident doc: what you observed, what you hypothesised, what you + checked, what it turned out to be. Include screenshots. +4. When you think you've found them all, run `make chaos-clean`, enable them one at + a time, and confirm each signature. +5. *Then* read `docs/PATHOLOGIES.md` and score yourself. +6. Repeat once with a new random set. The second run is where the method actually + becomes yours. + +**Done when:** you find at least 4 of the enabled pathologies from telemetry alone, +and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague. + diff --git a/docs/roadmap/phase-9.md b/docs/roadmap/phase-9.md new file mode 100644 index 0000000..c894b45 --- /dev/null +++ b/docs/roadmap/phase-9.md @@ -0,0 +1,161 @@ +# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)* + +**Status:** Not started + +**Message:** Not technically blocked by earlier phases (it stands on its own +as a judgment exercise), but sequenced last-but-one so Phase 8's incident to +re-diagnose in Kibana already exists. + +**Time-box:** 1–2 evenings. + +Babak's remaining questions are all *judgment* questions. He isn't testing whether +you can recite what Filebeat does; he's testing whether you have a framework for +choosing. Here's the framework, then the answers as worked examples. + +### The framework — six axes to reason on + +Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through: + +1. **Where does the data originate?** Inside your application (you can instrument it) + or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't + touch, syslog from a switch)? You can only use OTel SDKs where you own the code. +2. **What shape is it already in?** Structured JSON at the source, or unstructured + text needing grok/regex parsing? Parsing cost has to land *somewhere*. +3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate? +4. **What's the destination, and how many?** One backend forever, or several, or one + you expect to change? +5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent + on every host) distributes cost and adds resilience but multiplies config surface. + Parsing centrally concentrates cost and creates a single point of failure — but + one place to change rules. +6. **What already exists, and what does it cost?** Existing Elastic investment, + existing licences, team skills, and licensing constraints (Elastic's Filebeat and + Logstash are source-available under Elastic licensing; the OTel Collector, Fluent + Bit and Vector are permissively licensed). Migration is never free. + +### The answers + +**When to use Filebeat.** When you need a lightweight shipper that tails log *files* +from a host or container and forwards them with minimal transformation — and your +destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once +delivery with a registry of file offsets, backpressure handling, and prebuilt modules +for common services (nginx, MySQL, system logs) that come with parsing and dashboards +included. Reach for it when the source is a file you don't control the format of, the +app can't be modified, and Elasticsearch is where it's going. + +**When to use Logstash.** When you need real *transformation*: parsing unstructured +text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB), +splitting or merging events, conditional routing to multiple destinations, or +buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine +for events. It's heavyweight — a JVM — so you run a few of them centrally, not one +per host. + +**When to use Filebeat and Logstash together, and why.** This is the classic Elastic +architecture and the reason is a **division of labour between edge and centre**: + +- Filebeat runs on *every* host: cheap, tails files, handles offsets and + backpressure, ships onward. You want the thing running in a thousand places to be + small and boring. +- Logstash runs on a *few* central nodes: does the expensive grok parsing, + enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so + you want them in a handful of places you can redeploy, not on a thousand hosts. +- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's + persistent queue absorbs it and Filebeat backs off gracefully, rather than every + host queuing independently or dropping. + +So: Filebeat for *collection and delivery*, Logstash for *processing and routing*. +Use both when you have many hosts **and** non-trivial parsing/enrichment **and** +need central buffering. Skip Logstash when your logs are already structured JSON and +Elasticsearch ingest pipelines can do the little parsing that remains. + +**When to use OpenTelemetry.** + +- You own the application code and want telemetry generated *from inside it* — spans + with business context, custom metrics, logs already carrying trace IDs. +- You need **more than one signal**, correlated. This is the strongest argument: one + SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs + threaded through all three. Neither Filebeat nor Logstash does traces. +- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent + data model across all of them. +- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for + Jaeger by editing Collector YAML. +- You need distributed tracing at all. There is no serious vendor-neutral alternative. +- Your telemetry needs processing in flight — redaction, tail sampling, attribute + rewriting — without touching the app. + +**When *not* to use OpenTelemetry.** This is the question that separates people who +have used it from people who have read about it. + +- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary, + an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a + file-based collector. +- **You only need one signal from one source with no correlation.** A single nginx + access log going to a single log store: OTel is more machinery than the job needs. + Fluent Bit or Filebeat is simpler and finishes the job today. +- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature + Elastic estate with hundreds of Beats modules, dashboards and detection rules is + years of investment. "We should use OTel because it's the standard" is not a + business case. Note the direction of travel though: the Collector can already + receive from and export to most of these, so a *gradual* migration is usually the + right shape — often by putting a Collector or Vector in front of the existing + agents rather than ripping them out. +- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets, + where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes. +- **The signal is genuinely business analytics, not telemetry.** Clickstream data for + a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different + retention, different query patterns, different cost model. +- **Maturity gaps for a specific need in a specific language.** OTel logs support has + matured at different rates per language, and profiling is still alpha. Check the + status page for your language before betting on a signal. +- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks + on it. A two-person team that needs error tracking by Friday should install Sentry. + +The senior framing to give Babak: *these tools aren't competitors so much as +different layers.* OpenTelemetry is a **generation and transport standard** for +telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you +don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and +Elasticsearch are **stores**. Most real architectures use one from each layer, and +the interesting question is always which layer a given problem lives in. + +### Resources + +- Better Stack — *Filebeat vs Logstash: How to Choose* (free article). +- SigNoz — *OpenTelemetry vs ELK Stack* (free article). +- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* — + read these to see how the Collector does the Filebeat/Logstash job itself. +- OpenTelemetry docs → *Instrumentation status by language* — for the maturity + caveats. + +### Build — and this is no longer hypothetical + +Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml` +deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with +Filebeat reading Docker container logs and Logstash parsing JSON into daily +`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these +questions — the codebase he handed you contains both worlds. + +1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs** + flow down two pipelines: Filebeat → Logstash → Elasticsearch, and + Alloy → Loki. Compare them on the same data. +2. Test the direct ingest path the README documents — pipe a JSON line into + Logstash's TCP input on 5000, then query Elasticsearch — so you understand where + parsing happens before you change it. +3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter + that parses them. Then do the same job with Alloy's `loki.process` stages. Time + both. Note which one you'd rather debug at 2 a.m. +4. Compare the *investigation experience* directly: take one incident you already + diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that + you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana + wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL + for some questions, and Loki's cost model genuinely beats Elasticsearch's for + others. +5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic + licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are + permissively licensed. For some employers that alone decides it. +6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most + valuably — what you actually found annoying about each. Having run both stacks + against the same application is the difference between reciting a comparison and + arguing one. + diff --git a/docs/roadmap/summary.md b/docs/roadmap/summary.md new file mode 100644 index 0000000..d115381 --- /dev/null +++ b/docs/roadmap/summary.md @@ -0,0 +1,71 @@ +# Roadmap summary — all phases at a glance + +*Part of `docs/roadmap/` — the one-page overview. Read [`intro.md`](intro.md) +first for platform/background context; read the individual `phase-N.md` file +for the phase you're actually working on. This file exists so an agent (or +you) can check overall status and sequencing without loading every phase's +full detail.* + +Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase +file follows the same shape: **Status**, **Message** (what's done, notes, +warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. + +## Status and focus + +| Phase | File | Status | Focus | Time-box | Main output | +|---|---|---|---|---|---| +| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` | +| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations | +| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo | +| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | +| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation | +| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars | +| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` | +| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles | +| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts | +| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` | +| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` | +| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo | + +## What each phase is, in one line + +- **0 — Build the base project, test-first.** Write the black-box test suite, + then implement the seven-service system against it as a frozen contract. + **Done** — see `phase-0.md` for the full implementation report. +- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector + split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via + Ansible and get one auto-instrumented trace into Tempo. +- **2 — Distributed tracing.** The heart of the task: auto- and + manual-instrument all seven services, and hand-propagate trace context + across the Redis Stream hop, which carries none by default. +- **3 — Structured logs, correlated to traces.** Replace prose logging with + structured events carrying a lowercase `trace_id`, so Loki and Tempo link + in both directions. +- **4 — Metrics.** Instrument RED metrics per service, understand cardinality + limits, and route app metrics to Prometheus given Alloy drops OTLP metrics + silently. Tempo's metrics-generator already produces span metrics and the + service graph — don't duplicate it in the collector. +- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the + Grafana Foundation SDK and ship it through the Ansible role, so "destroy + and recreate in seconds" is a real, repeatable command. +- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't + be averaged, what coordinated omission is, and drive Ledgerline with + arrival-rate executors to get a clean before/after load profile. +- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and + SLOs from the Phase 6 baseline (not from thin air), write an error-budget + policy, and wire multi-burn-rate alerts. +- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set + of chaos flags, find the pathologies from telemetry alone using a top-down, + hypothesis-driven method, and write it up as an incident doc. +- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs + OpenTelemetry on six concrete axes, using the platform's real second stack + (Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint. +- **10 — Package the delivery.** README, a rehearsed five-minute demo, the + written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`, + `CONCEPTS.md`), and an honest "what I'd do differently in production" + section. + +## If you have to cut scope + +Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not +cut Phase 8** — it's the only phase that proves you learned anything.