Add project documentation and Claude instructions

Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
2026-09-05 08:11:00 +03:30
commit eb99cdbb8c
19 changed files with 3372 additions and 0 deletions
@@ -0,0 +1,9 @@
 __pycache__/
 *.pyc
 *.pyo
 .pytest_cache/
 .env
 *.egg-info/
 .DS_Store
 venv/
 .venv/
@@ -0,0 +1,398 @@
 # CLAUDE.md — Ledgerline
 Project instructions for Claude. Read this fully before acting.
 ---
 ## What this project is
 **Ledgerline** is a deliberately imperfect seven-service Python/FastAPI system
 simulating a payments backend. The business logic is fake and does not matter.
 The project exists as a **subject for learning observability**. The real
 deliverable is not working software — it is the user's understanding of
 OpenTelemetry instrumentation, the Grafana stack (Tempo / Loki / Prometheus /
 Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is
 assessed by a mentor, Babak.
 The full plan lives in `docs/roadmap/`, split one file per phase so an agent
 working on a given phase doesn't have to load the other ten: `intro.md`
 (everything before Phase 0 — the platform, the resource spine, background
 reading), `phase-0.md` through `phase-10.md`, and `summary.md` (a one-page
 status/sequencing overview of all of them). Phase 0 is built test-first from
 two specs: `docs/phase-0a-test-suite-spec.md` (the tests) and
 `docs/phase-0-base-project-spec.md` (the system).
 ### The observability backend is external and pre-built
 The Grafana stack is **not** built in this repo. It is deployed by a separate
 Ansible project, `observability-ansible`, which the user received from his mentor.
 It deploys two stacks on a target host via Docker Compose:
 - **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with
  **Grafana Alloy** as the collector and node-exporter for host metrics.
 - **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the
  hands-on subject of Phase 9, not part of the main pipeline.
 Five facts about that platform that change what correct code looks like here. Verify
 them against the live system rather than assuming — they were read from the configs
 at one point in time and the user may have since extended them.
 1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
   `traces` and nothing else. OTLP **metrics and logs are accepted and silently
   discarded** — no error anywhere. This is deliberate: the platform's own diagram
   labels the OTLP input "spans + resource attrs" and draws app metrics as a
   separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
   metrics "should be arriving"; check Prometheus or the Alloy UI first.
 2. **App metrics reach Prometheus by direct scrape, not through Alloy.** Alloy has
   no `prometheus.scrape` component; `prometheus.yml.j2` scrapes the stack's own
   components plus whatever is listed in `prometheus_extra_scrape_jobs`. Note that
   the diagram draws this differently (app metrics routed via Alloy) — the diagram
   and the code have drifted, and the user knows. Don't "correct" one to the other
   unasked.
 3. **Span metrics and the service graph come from Tempo's metrics-generator**
   (`processors: [service-graphs, span-metrics, local-blocks]`, remote-writing to
   Prometheus with `send_exemplars: true`) — **not** from collector connectors.
   Never add `spanmetrics` / `servicegraph` to Alloy; it duplicates every series.
   The metric names differ from the ones in most blog posts — look them up in
   Prometheus before writing PromQL.
 4. **Log→trace correlation runs on a case-sensitive regex:**
   `trace[_-]?id["=: ]+([a-fA-F0-9]+)`. Logs must carry **lowercase `trace_id`**.
   OpenTelemetry's logging auto-instrumentation emits `otelTraceID`, which does not
   match. Exemplar labels must also be `trace_id`. The diagram names this outright:
   "Correlation key: trace_id / exemplars."
 5. **There is no dashboard provisioning in the platform** — datasources only.
   Dashboards-as-code is the user's deliverable and belongs in the Ansible role, so
   that `ansible-playbook` rebuilds them.
 Ledgerline talks to this platform over the network. Endpoints come from env
 (`OTEL_EXPORTER_OTLP_ENDPOINT` etc.), never hardcoded.
 ### The single most important thing to understand
 **The application is broken on purpose.** It contains thirteen deliberate
 pathologies — a missing index, a starved connection pool, an N+1 query, a
 blocking call inside an `async def`, a retry storm with no backoff, an unbounded
 cache, and more. They are documented in `docs/PATHOLOGIES.md` and toggled at
 runtime via Redis-backed chaos flags.
 **Never fix them. Never suggest fixing them. Never "helpfully" clean them up
 while doing something else nearby.** Seeing `pool_size=2` or a `time.sleep()`
 in an async handler and correcting it destroys weeks of the user's work.
 If a pathology genuinely blocks something, say so and ask — don't act.
 ---
 ## Spoiler rules — hard constraints
 These override helpfulness. Breaking any of them ruins the exercise.
 1. **Do not read, summarise, quote, paraphrase, or hint at the contents of
   `docs/PATHOLOGIES.md`** unless the user explicitly says they have finished
   Phase 8 and wants to review it. Not even "let me check the docs to confirm."
 2. **Do not reveal which chaos flags are currently enabled** during Phase 8. Do
   not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and
   do not infer the set from the source and announce it. If the user asks you to
   check, remind them what phase they're in and let them insist — then comply.
 3. **Do not run `pytest -m pathology` during Phase 8.** The pathology suite has
   one named test per flag, so its pass/fail output is a complete answer key.
   Same for `make test-pathology` and `make test-slow`. Running the `contract`,
   `seed`, `baseline` or `guard` markers is fine.
 4. **Do not diagnose the system for the user during Phase 8.** That phase is the
   capstone: they find the bottlenecks from telemetry alone. Your job there is
   to teach *method* (how to read a trace gap, how to reason about saturation)
   and to operate tools they ask you to operate — not to name the culprit. If
   they're genuinely stuck after a real attempt, see the last Teaching Mode
   bullet: help directly rather than fishing.
 5. Outside Phase 8 these relax. Ask which phase they're in if it's unclear.
 ---
 ## Current phase
 > **Phase: 0a** — writing the integration test suite.
 > *(The user updates this line. If it looks stale relative to the conversation,
 > ask before assuming.)*
 Phase reference: **0a** test suite · **0b** implement to green · 1 concepts +
 first trace · 2 tracing · 3 logs · 4 metrics · 5 dashboards-as-code ·
 6 k6 + percentiles · 7 SLO/error budget · 8 bottleneck hunt ·
 9 Filebeat/Logstash/OTel judgment · 10 delivery.
 ---
 ## Teaching Mode
 *Portable rule — copy this section into any project's `CLAUDE.md` as-is.*
 When the user asks to be *taught* a piece of work rather than have it done
 for them (building a config file, writing a script, running a sequence of
 commands, standing up some infrastructure), switch out of the default
 "just complete the task" workflow and into this one instead. Stay in it
 for the rest of that work item unless the user opts out.
 - **Never produce the finished artifact yourself.** Break it into small,
  sequential checkpoints instead of one big deliverable. Introduce one
  checkpoint at a time — what it needs to do and why — then stop and wait
  for the user's own attempt before moving to the next one.
 - **Teach the concept before asking for an attempt**, whenever a checkpoint
  depends on something the user hasn't demonstrated familiarity with yet.
  Ground the explanation in the actual thing being built, not in the
  abstract — use the user's own file/system as the running example.
 - **Verify their actual work, not their claim about it.** When they say
  "I did X" or "check this," go read the file / inspect the live state
  yourself before responding. Don't take it on faith.
 - **When something's wrong, give why + how + how to verify — never just
  the fix.** State the underlying mechanism that makes it wrong (not just
  "this is incorrect"), point at what needs to change conceptually so the
  user writes the correction themselves, and show them how to confirm the
  fix actually worked once they've made it.
 - **Prefer real verification over asserted theory, for you as much as for
  them.** When you're not fully certain how something will actually
  behave — or even when you are, if it's cheap to check — run it for real
  against the live system rather than describing what "should" happen. If
  a prediction turns out wrong once verified, say so plainly and correct
  course instead of quietly rewriting the story to fit.
 - **Use scratch/throwaway resources for exploratory experiments** that
  aren't part of the user's real deliverable (settling a "what actually
  happens if..." question), and clean them up afterward so they don't
  pollute the real project state. Ask before deleting anything that might
  be real work rather than obvious scratch output.
 - **Stop withholding once the user is genuinely stuck.** If they say "I
  don't know" or make a real attempt and can't get further, answer
  directly instead of continuing to fish for guesses — the goal is
  understanding, not making them struggle for its own sake.
 - **When the user proposes their own explanation for confusing behavior,
  test it rather than defaulting to your own theory.** Treat it as a real
  hypothesis worth verifying, not something to talk them out of.
 ---
 ## Teaching Mode scope in this project
 Teaching Mode is the **default** here. But not everything in this repo is
 learning material, and treating scaffolding as a lesson wastes the user's time.
 Decide by what's being touched:
 | Path / concern | Mode |
 |---|---|
 | `services/**` business logic, `scripts/seed.py`, `db/schema.sql`, frontend, Compose plumbing, Makefile | **Just build it.** This is scaffolding. The user has stated he doesn't care about the microservices themselves. |
 | `tests/**` | **Just build it** during Phase 0a, from the test spec. Afterwards it is a **frozen contract** — see the rules below. |
 | `common/telemetry.py`, any OTel SDK code inside services, span/metric/log instrumentation | **Teaching Mode.** |
 | `observability/**` — Collector config, Grafana provisioning, datasource cross-links, dashboards-as-code, alert rules | **Teaching Mode.** |
 | `loadtest/**` — k6 scripts, executors, thresholds | **Teaching Mode.** |
 | `docs/SLO.md`, `docs/INVESTIGATION.md`, `docs/PIPELINE-CHOICES.md`, `docs/CONCEPTS.md` | **Teaching Mode** — these are the user's own reasoning, written in his own words. Never draft them for him. Review and challenge instead. |
 | PromQL / LogQL / TraceQL queries | **Teaching Mode.** Query languages are a skill, not a fact. |
 **Phase 0 is explicitly exempt**, in both halves. The base project was delegated
 to coding agents by design: **0a** writes the tests, **0b** implements against
 them. Build both from their specs without teaching. Phases 1–10 are the learning
 work.
 If the user says "just do it," "give me the code," or "skip the socratic
 thing" — comply immediately and without negotiation for that work item.
 ### The test suite is a contract
 Phase 0 is built test-first by two separate agents, and the whole value of that
 split collapses if the tests drift toward the implementation.
 - **During 0a:** write only `tests/`. Do not create stubs, scaffolding, or
  placeholder modules under `services/`, `common/`, `db/` or `scripts/` — not
  even to make imports resolve. Tests talk to the system over HTTP, Postgres,
  Redis and the Docker CLI, never by importing it.
 - **During 0b and afterwards:** `tests/` is **read-only**. Never edit, delete,
  skip, `xfail`, or loosen a test, a threshold, a marker, or a fixture to make
  code pass. If a test looks wrong or impossible, **stop and say so** — the user
  decides, not you.
 - **Never fake a pathology to satisfy a timing assertion.** A `sleep()` tuned to
  clear a latency ratio is a failure even when the suite goes green. Implement
  the mechanism described in the Phase 0 spec §7. Several assertions (query
  counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception
  paths) are deliberately built so faking is harder than implementing; treat that
  as the intent for all of them.
 - From Phase 3 onward the log-line-counting assertions in the pathology suite
  *should* be rewritten against structured logs and then span/metric data. That
  is a planned migration the user drives — propose it, don't do it unasked, and
  it does not license loosening anything else.
 ---
 ## Calibrating explanations to this user
 Senior Python engineer, 7+ years. Django, FastAPI, PostgreSQL, SQLAlchemy,
 Celery, Docker, AWS/GCP, Terraform, Ansible, Kubernetes. Runs his own
 single-node cluster.
 - **Don't explain:** Python, async/await, FastAPI, Pydantic, SQLAlchemy, SQL,
  `EXPLAIN ANALYZE`, Docker, Compose, HTTP, REST, git, Linux.
 - **Do explain:** the OpenTelemetry data model and API/SDK/Collector split,
  context propagation mechanics, Prometheus's series/cardinality model, PromQL,
  LogQL, TraceQL, Loki's label vs body distinction, span metrics and service
  graph connectors, exemplars, k6 executors and coordinated omission, SLI/SLO
  arithmetic and burn rates.
 - Connect new concepts to things he already knows where it genuinely helps —
  `contextvars` for in-process trace context, connection-pool queueing for
  saturation, `EXPLAIN ANALYZE` for what a DB span can't show you.
 - He learns from professional reference implementations. When a pattern is
  non-obvious, point him at the relevant file in the OpenTelemetry Demo
  (`open-telemetry/opentelemetry-demo`) rather than inventing an example.
 - He extracts study notes from video subtitles. When recommending video, note
  whether captions exist.
 ---
 ## Verification — check the live system, don't assert
 Teaching Mode's "verify their actual work" and "prefer real verification"
 bullets are not optional here. Concrete ways to do it in this stack (confirm
 ports against `docker-compose.yml` — these are the intended defaults):
 ```bash
 docker compose ps                      # what's actually running
 make smoke                             # end-to-end sanity
 docker compose logs -f gateway         # actual log shape, not the intended one
 make test-contract                     # fast functional suite (< 60s)
 make test-baseline                     # is the clean system actually healthy
 pytest tests/ -m guard -q              # "no observability yet" — delete at Phase 2
 git diff --stat tests/                 # did anything touch the contract? should be empty
 ```
 The pathology suite (`make test-pathology`, `make test-slow`) is slow and is an
 answer key — see spoiler rule 3 before running it.
 **Traces (Tempo, :3200)** — search by service and duration rather than trusting
 that a span exists:
 ```bash
 curl -s 'http://localhost:3200/api/search?tags=service.name%3Dledger&limit=5'
 curl -s "http://localhost:3200/api/traces/<trace_id>" | jq '.batches | length'
 ```
 **Metrics (Prometheus, :9090)** — and this is the cardinality check worth
 running often:
 ```bash
 curl -s 'http://localhost:9090/api/v1/query?query=count({__name__=~"ledgerline.*"})'
 curl -s 'http://localhost:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'
 ```
 **Logs (Loki, :3100)** — confirm `trace_id` is actually in the payload, not just
 in the code that was supposed to put it there:
 ```bash
 curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="gateway"} | json | trace_id != ""' \
  --data-urlencode 'limit=5'
 ```
 **Alloy (:12345)** — the component graph is the fastest way to see what the pipeline
 actually does, including which signal outputs are wired. Check it before theorising
 about missing data.
 **Metric names** — never assume; the Tempo-generated names differ from blog posts:
 ```bash
 curl -s 'http://<obs-host>:9090/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))'
 ```
 **Grafana (:3000)** — verify provisioning actually loaded rather than reading
 the YAML and assuming:
 ```bash
 curl -s -u admin:admin 'http://localhost:3000/api/datasources' | jq '.[].name'
 curl -s -u admin:admin 'http://localhost:3000/api/search?type=dash-db' | jq '.[].title'
 ```
 Rules of thumb:
 - After the user says "I added the span" / "the exporter is configured" — go
  look at the file *and* query the backend. Code present ≠ data arriving. The
  gap between those two is where most of the real learning is.
 - Before predicting how the Collector, a sampler, or a PromQL query will behave:
  if it's cheap to run, run it. Batching, sampling and cardinality behaviour are
  all easy to get wrong from memory.
 - If a prediction turns out wrong, say so plainly and correct course.
 ---
 ## Scratch resources
 For "what actually happens if…" experiments that aren't part of the deliverable:
 - Scratch dashboards → prefix `scratch-`, uid prefix `scratch-`.
 - Scratch metrics → prefix `scratch_`.
 - Scratch Compose services → `observability/scratch.compose.yml`, never the main
  file.
 - Scratch scripts → `/tmp/`, not the repo.
 Clean up afterwards. **Never delete** anything under `observability/`,
 `loadtest/`, `docs/`, or any dashboard without a `scratch-` prefix without
 asking first — a hand-tuned dashboard or a half-written investigation doc is
 real work even if it looks messy.
 ---
 ## Anti-patterns specific to this project
 Things a well-meaning assistant does wrong here:
 - **Fixing pathologies.** Adding an index, raising `pool_size`, wrapping the
  blocking call in `to_thread`, adding backoff to the retry loop. Never.
 - **Adding observability during Phase 0.** The base app ships with plain-text
  stdlib logging and a weak manual `X-Request-Id`. That's the "before" state
  the whole project measures against. `tests/test_no_observability.py` enforces
  it — if that test fails, something crept in; don't delete the test to fix it.
 - **Editing a test to make code pass.** Including "just relaxing the threshold,"
  adding `@pytest.mark.skip`, or widening an assertion because a run was noisy.
  Report it instead.
 - **Satisfying a pathology test with a `sleep()`** rather than the real
  mechanism.
 - **Importing application code into `tests/`.** The suite is black-box on
  purpose; an import couples it to implementation and the contract stops meaning
  anything.
 - **Writing `common/telemetry.py` for him** in Phase 2. That file is the single
  most valuable thing he writes. Checkpoint it: Resource → TracerProvider →
  BatchSpanProcessor → exporter → propagator, one at a time.
 - **Creating or editing dashboards through the Grafana UI or HTTP API** as a
  deliverable. Everything must come from provisioning files or the Foundation
  SDK generator, because "destroy and recreate in seconds" is an explicit
  requirement from Babak. Read-only API calls for *verification* are fine.
 - **Writing OpenTelemetry Collector YAML.** The platform uses **Alloy**, whose
  config is component-based (`otelcol.receiver.otlp`, `otelcol.exporter.otlp`,
  `loki.source.docker`, `prometheus.remote_write`), not YAML receiver/processor/
  exporter pipelines. A correct-looking Collector config is a wrong answer here.
 - **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
  already does this.
 - **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in
  Prometheus / Loki before claiming anything works.
 - **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
  Loki derived field with no visible error.
 - **`avg(p99)` or averaging any quantile** across instances or time buckets.
  Aggregate the histogram buckets, then compute the quantile.
 - **High-cardinality metric labels** — `account_id`, `transaction_id`,
  `merchant_name`, raw paths. These belong on spans. (There is one *deliberate*
  cardinality-explosion exercise in Phase 4; that one is intentional and gets
  reverted.)
 - **`constant-vus` for load profiles** where an open model is correct.
  Closed-model executors hide latency through coordinated omission.
 - **Putting `trace_id` in a Loki label** instead of the log body + derived field.
 - **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a
  Phase 9 judgment exercise and a small side pipeline only — not the
  architecture.
 - Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md`
  for him. Those are his reasoning. Review, challenge, poke holes — don't write.
 ---
 ## Conventions
 - Python 3.12, FastAPI, SQLAlchemy 2.0 async + asyncpg, `httpx` async,
  `redis-py` asyncio, Pydantic v2. No Celery, no RabbitMQ — Redis Streams,
  deliberately, because it has no automatic context propagation.
 - One Docker image for all services; `CMD` differs per service.
 - Semantic conventions for standard attributes (`http.request.method`,
  `db.system`, `server.address`). Namespace custom ones as `ledgerline.*`.
 - `service.name` matches the directory name under `services/`.
 - Duplication across services is fine. Don't refactor toward a shared framework.
 - Docker Compose only, locally, through Phase 9. Kubernetes is a later
  follow-up — porting instrumentation and debugging a cluster at the same time
  is how this stalls.
 - Never save a Grafana dashboard from the UI.
@@ -0,0 +1,371 @@
 # Ledgerline
 A deliberately imperfect seven-service payments backend, built as a subject for
 learning observability with OpenTelemetry and the Grafana stack.
 > ### ⚠️ This system is broken on purpose
 >
 > Ledgerline ships with thirteen intentional performance and reliability
 > pathologies: a missing index, a starved connection pool, an N+1 query, a
 > blocking call inside an `async def`, a retry storm with no backoff, an
 > unbounded cache, and more. They are toggled at runtime and documented in
 > [`docs/PATHOLOGIES.md`](docs/PATHOLOGIES.md).
 >
 > **Do not fix them.** They are the entire point. If you are here to contribute,
 > read [`CLAUDE.md`](CLAUDE.md) first.
 The business logic is fake and does not need to be correct, secure, or
 defensible. What it needs to be is *structurally* realistic: real network hops,
 real SQL, real async message passing, real contention, and real latency
 distributions — so that instrumenting it teaches something true.
 ---
 ## Quickstart
 Requires Docker and Docker Compose. Nothing else.
 ```bash
 git clone <this-repo> && cd ledgerline
 cp .env.example .env
 make up             # start postgres, redis, and the seven services
 make seed           # schema + 250k transactions across 800 accounts (~60s)
 make smoke          # end-to-end sanity check
 make test-contract  # fast functional suite (~60s)
 ```
 Then open **http://localhost:8000**.
 The observability backend lives in a separate Ansible repo and is deployed to its
 own host. From Phase 1 onward:
 ```bash
 # in observability-ansible/
 ansible-playbook playbooks/grafana-stack.yml
 ```
 Then point Ledgerline at it via `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`.
 Reset everything, including volumes:
 ```bash
 make reset
 ```
 ---
 ## Architecture
 ```mermaid
 flowchart LR
    B[browser] --> GW[gateway :8000]
    GW --> LD[ledger :8001]
    GW --> FR[fraud :8002]
    FR --> SA[sanctions :8003]
    GW --> IN[insights :8004]
    GW -- XADD --> ST[(redis stream)]
    ST --> NO[notifier :8005]
    ST --> AU[audit worker]
    LD --> PG[(postgres)]
    SA --> PG
    IN --> PG
    NO --> PG
    AU --> PG
    IN --> RD[(redis cache)]
 ```
 | Service | Port | Role | Stores |
 |---|---|---|---|
 | `gateway` | 8000 | Public API, serves the frontend, orchestrates everything | Redis (flags) |
 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow vendor" | Postgres (read) |
 | `insights` | 8004 | 120-day analytics over transaction history | Postgres + Redis |
 | `notifier` | 8005 | HTTP API **and** a Redis Stream consumer | Postgres |
 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |
 Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz`
 (dependency check).
 **Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no
 automatic trace-context propagation, so Phase 2 has to implement it by hand.
 That's the most valuable exercise in the project. Don't swap it out.
 ### The primary request flow
 `POST /api/transactions` produces a trace roughly four levels deep with one
 async fork:
 ```
 gateway
  ├─ ledger    create transaction (pending, row lock on account)
  ├─ fraud     score
  │    └─ sanctions   watchlist lookup
  ├─ ledger    settle or reject
  ├─ redis     XADD stream:transactions  ──┬─ notifier  (consumer group)
  │                                        └─ audit     (consumer group)
  └─ insights  cached account summary
 ```
 The second flow, `GET /api/accounts/{id}/insights`, is a read-heavy aggregation
 and behaves very differently under load.
 ---
 ## Chaos flags
 Pathologies are controlled at runtime via a Redis hash (`chaos:flags`), read on
 every request — **no restart needed**. Toggle them from the frontend's chaos
 panel, the API, or the Makefile.
 ```bash
 make chaos-clean       # all off — the baseline
 make chaos-light       # a mild, realistic amount of pain
 make chaos-nightmare   # all thirteen on
 make chaos-random      # 3–6 flags, seed never logged
 ```
 ```bash
 curl -s localhost:8000/api/admin/chaos | jq
 curl -sX POST localhost:8000/api/admin/chaos \
  -H 'content-type: application/json' \
  -d '{"flag":"slow_sanctions","enabled":true}'
 ```
 Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`,
 `pool_starvation`, `row_lock_contention`, `blocking_event_loop`, `retry_storm`,
 `no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`,
 `chatty_health`.
 > **During Phase 8, don't open the chaos panel and don't call
 > `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the
 > seed; the point is to find the pathologies from telemetry alone. The panel
 > will happily tell you the answer, which is the one thing you don't want.
 ---
 ## Make targets
 | Target | Effect |
 |---|---|
 | `up` / `down` | Start / stop the application stack |
 | `logs` | `docker compose logs -f` |
 | `seed` | Apply schema and seed 250k transactions |
 | `reset` | `down -v` → `up` → `seed` |
 | `smoke` | One transaction + one insights call, assert 200s |
 | `test` | Full suite except the slow markers |
 | `test-contract` | Functional + seed + guard assertions, < 60 s |
 | `test-baseline` | Is the clean system genuinely healthy |
 | `test-pathology` | One differential test per pathology — ⚠️ answer key, see below |
 | `test-slow` | The soak-ish pathology tests |
 | `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile |
 | `fix-index` / `drop-index` | Add or remove the deliberately missing indexes |
 | `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 5+* |
 | `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 6+* |
 ---
 ## Testing
 Phase 0 was built **test-first, by two separate agents**: one wrote `tests/`
 from `docs/phase-0a-test-suite-spec.md` without seeing any implementation, the
 second implemented the services until the suite went green without being allowed
 to touch a test.
 The suite is **black-box**. It imports nothing from the application and talks to
 the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals,
 not mechanics — there are no references to function names, module paths, or SQL
 strings anywhere in it.
 Two halves:
 - **Working state.** A transaction really traverses all seven services (stop
  `sanctions` and transactions must fail — that's the test that catches a
  collapsed monolith); async work really happens off the request path via two
  independent consumer groups; the seed data is really 250k rows, really skewed
  toward five hot accounts, and really contains all five varieties of poison row.
 - **Broken state.** One differential test per pathology, asserting the system
  measurably degrades when a flag is on versus off. Assertions favour signals
  that are hard to fake: `pg_stat_statements` query counts for the N+1,
  `EXPLAIN` plan node types for the missing index, Redis consumer-group lag for
  the slow consumer, container RSS for the memory leak, throughput plateaus for
  pool starvation. A `sleep()` won't satisfy those.
 ```bash
 make up && make seed      # prerequisite — the suite runs against a live stack
 make test-contract        # fast
 make test                 # everything except slow
 ```
 > ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so
 > its output is a complete answer key. Don't run it during the Phase 8
 > bottleneck hunt.
 `tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no
 `prometheus_client`, no structured logging, no `/metrics`. **Delete it at the
 start of Phase 2**; it has done its job by then.
 Some pathology assertions currently count log lines, because plain text is the
 only cross-service observability surface that exists at Phase 0. That is the
 deficiency this project sets out to fix — those assertions get rewritten against
 structured logs in Phase 3 and span data in Phase 4, and become both simpler and
 more robust. See `tests/README.md`.
 ---
 ## Ports
 | | Port | |
 |---|---|---|
 | Frontend / gateway | 8000 | |
 | `ledger` … `notifier` | 8001–8005 | |
 | Postgres | 5432 | `ledgerline` / `ledgerline` |
 | Redis | 6379 | |
 On the observability host (deployed by `observability-ansible`):
 | | Port | |
 |---|---|---|
 | Grafana | 3000 | credentials from `group_vars/all.yml` |
 | Prometheus | 9090 | remote-write receiver enabled |
 | Alertmanager | 9093 | |
 | Loki | 3100 | |
 | Tempo | 3200 | |
 | Alloy UI | 12345 | component graph — check here first when data is missing |
 | OTLP (Alloy) | 4317 / 4318 | gRPC / HTTP |
 | Node Exporter | 9100 | |
 ---
 ## Observability stack
 The backend is **not** part of this repo. It is deployed by a separate Ansible
 project, `observability-ansible`, onto a target host:
 ```
 Ledgerline /metrics ──────scrape──────────────┐
 container stdout ──docker.sock──▶ Alloy ──▶ Loki
 Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo
                                              │  span metrics + exemplars
                                              ▼         (remote_write)
                                         Prometheus ◀────────┘
                                              │
                                       Alertmanager ──▶ Receivers
                                              │
   Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above
          Correlation key: trace_id / exemplars
 ```
 RED metrics and the service graph are generated by **Tempo's metrics-generator**,
 not by collector connectors — do not add `spanmetrics`/`servicegraph` to Alloy.
 Three constraints inherited from that platform, worth knowing before you debug
 anything:
 - **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
  accepted and silently dropped — by design; the platform's diagram labels that
  input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
  each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
 - **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
  case-sensitively.** Log lines must carry lowercase `trace_id`. OTel's logging
  auto-instrumentation emits `otelTraceID` and will not match.
 - **The platform provisions datasources only.** Every dashboard here is generated
  from code with the Grafana Foundation SDK and shipped through the Ansible role,
  so `ansible-playbook` rebuilds them from scratch. **Never save a dashboard from
  the Grafana UI** — it won't survive the next run.
 Set `OTEL_EXPORTER_OTLP_ENDPOINT` (and friends) in `.env` to point at the Alloy
 host. Nothing observability-related is hardcoded.
 ---
 ## The five-minute demo
 Rehearse this before showing anyone.
 1. Delete every dashboard in the Grafana UI, then re-run
   `ansible-playbook playbooks/grafana-stack.yml` — platform and dashboards both
   rebuilt from code, one command.
 2. Submit one transaction from the frontend. Follow that single request through
   Grafana: RED dashboard → latency spike → exemplar → trace → the slow span →
   that span's logs → back to the trace. One request, three signals, no
   searching.
 3. `make chaos-nightmare && make load-stress` — watch p95 diverge from p50, the
   service map turn red at a specific node, the error budget burn, and the
   fast-burn alert fire.
 4. Walk through [`docs/INVESTIGATION.md`](docs/INVESTIGATION.md) — how the
   pathologies were found from telemetry alone.
 ---
 ## Repository layout
 ```
 ├─ CLAUDE.md                  # instructions for AI assistants — read before contributing
 ├─ Makefile
 ├─ docker-compose.yml
 ├─ Dockerfile                 # one image, all Python services
 ├─ db/
 │  ├─ schema.sql
 │  ├─ optional_indexes.sql    # the deliberately-omitted indexes
 │  └─ seed_manifest.json      # written by the seeder; the tests' source of truth
 ├─ scripts/                   # seed.py, smoke.py
 ├─ common/                    # settings, db, redis, http, logging, telemetry
 ├─ services/                  # gateway ledger fraud sanctions insights notifier audit
 │  └─ */chaos.py              # each service's pathologies, clearly marked
 ├─ tests/                     # black-box integration suite — a frozen contract
 │  ├─ harness/                # load driver, pg_stat helpers, stream + container probes
 │  └─ README.md               # methodology, markers, known-flaky list
 ├─ observability/             # dashboard generator (Foundation SDK); output ships via the Ansible role
 ├─ loadtest/                  # k6 scripts
 └─ docs/
   ├─ roadmap/                       # the ten-phase learning plan, one file per phase
   │  ├─ intro.md                    # platform background, resource spine (read first)
   │  ├─ phase-0.md … phase-10.md    # one phase per file — status, message, concepts, build
   │  └─ summary.md                  # one-page status/sequencing overview of all phases
   ├─ phase-0a-test-suite-spec.md   # spec the test suite was written from
   ├─ phase-0-base-project-spec.md  # spec the base app was built from
   ├─ PATHOLOGIES.md                # ⚠️ spoilers — the answer key
   ├─ SLO.md                        # SLIs, SLOs, SLA, RTO/RPO, error budget policy
   ├─ CONCEPTS.md                   # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words
   ├─ INVESTIGATION.md              # the Phase 8 bottleneck hunt, written as an incident doc
   └─ PIPELINE-CHOICES.md           # Filebeat vs Logstash vs OpenTelemetry — when and why
 ```
 ---
 ## Status
 - [ ] **Phase 0a** — integration test suite, written first, red against nothing
 - [ ] **Phase 0b** — base project implemented until the suite is green
 - [ ] **Phase 1** — concepts, first trace end to end
 - [ ] **Phase 2** — distributed tracing across all seven services
 - [ ] **Phase 3** — structured logs, correlated with traces
 - [ ] **Phase 4** — metrics, RED, service map, exemplars
 - [ ] **Phase 5** — dashboards and alerts as code
 - [ ] **Phase 6** — k6 load testing, percentiles
 - [ ] **Phase 7** — SLIs, SLOs, error budgets, RTO/RPO
 - [ ] **Phase 8** — bottleneck hunt
 - [ ] **Phase 9** — Filebeat / Logstash / OpenTelemetry judgment
 - [ ] **Phase 10** — delivery
 Full plan: [`docs/roadmap/`](docs/roadmap/) — start with
 [`intro.md`](docs/roadmap/intro.md), then the phase you're on, or
 [`summary.md`](docs/roadmap/summary.md) for the one-page overview.
 ---
 ## Non-goals
 No authentication, no authorization, no input hardening, no unit tests, no
 Kubernetes, no production-grade storage. Ledgerline runs on a laptop under
 Docker Compose and is never exposed to a network you don't control.
 The integration suite is deliberately the *only* test layer: it exists to pin
 down the system's observable behaviour — working and broken — not to verify the
 fake business logic.
 Things deliberately left for a later iteration, and worth naming when discussing
 the project: tail-based sampling instead of 100%, Mimir instead of raw
 Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII
 redaction in the Collector, and multi-tenancy.
@@ -0,0 +1,156 @@
 # Pathology catalogue
 Thirteen deliberate pathologies, toggled at runtime via the `chaos:flags` Redis
 hash (never a restart). Each lives in a `chaos.py` module inside its owning
 service. All flags default off.
 > Spoiler note (see root `CLAUDE.md`): this file is a complete answer key for
 > the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you,
 > until that phase is done.
 ---
 ## slow_sanctions (`sanctions`)
 **Mechanism:** `services/sanctions/chaos.py`. On every `/internal/lookup`
 call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of
 calls instead sleep a flat 5s -- longer than the 3s downstream timeout every
 caller uses (`common/http.py`), so these surface as timeouts one hop up, in
 `fraud`.
 **Symptom:** heavy-tailed latency on `POST /api/transactions`; most requests
 look normal, a fat p95/p99 tail; occasional 504s from `fraud`/`gateway` with
 no corresponding 5xx logged in `sanctions` itself (it never errors -- it's
 just slow).
 ## sanctions_errors (`sanctions`)
 **Mechanism:** `services/sanctions/chaos.py`. 3% of `/internal/lookup` calls
 return HTTP 503 outright, chosen independently per call via `random.random()`.
 **Symptom:** a small, steady rate of 502/504 responses from `gateway`, fully
 uncorrelated with load or account. `sanctions` logs a "lookup" line for every
 attempt regardless, so its request count won't match its success count.
 ## n_plus_one (`insights`)
 **Mechanism:** `services/insights/analytics.py::_n_plus_one_summary`. Instead
 of one `LEFT JOIN` against `merchant_categories`, fetches up to 500 raw
 transaction rows and issues one additional `SELECT ... FROM merchant_categories
 WHERE id = :id` per row.
 **Symptom:** `GET /api/accounts/{id}/insights` for a high-volume (hot) account
 issues 50-500+ SQL statements instead of 1-2, and its latency grows with the
 account's transaction count rather than staying flat.
 ## pool_starvation (`ledger`)
 **Mechanism:** `services/ledger/db.py`. Two engines are built at startup: a
 normal one (`pool_size=20, max_overflow=10`) and a starved one (`pool_size=2,
 max_overflow=0, pool_timeout=10`). Every `ledger` request picks which
 already-running pool to borrow a connection from based on the current flag --
 this is what makes the toggle live without a restart, since SQLAlchemy pools
 can't be resized in place.
 **Symptom:** throughput plateaus as concurrency increases past ~2 in-flight
 requests; latency and queueing rise sharply under concurrent load while a
 single request in isolation still looks fine.
 ## row_lock_contention (`ledger`)
 **Mechanism:** `services/ledger/repository.py::settle_transaction`. Only for
 accounts with `is_hot = true`: takes `SELECT balance_minor FROM accounts ...
 FOR UPDATE` and holds the transaction open for an extra 150ms before
 committing the balance update.
 **Symptom:** concurrent transactions against a hot account show p95 latency
 several times higher than the same concurrency spread across many cold
 accounts; cold-account traffic is unaffected.
 ## blocking_event_loop (`fraud`)
 **Mechanism:** `services/fraud/chaos.py::synchronous_cpu_work`. Runs ~120ms of
 synchronous SHA-256 hashing directly inside the `async def` handler for
 `/internal/score` -- no `run_in_executor`/`to_thread`.
 **Symptom:** concurrent requests to `fraud` (and anything waiting on it)
 serialise: wall-clock time for N concurrent requests approaches N × single-request
 time instead of staying close to single-request time, even though nothing is
 CPU-bound at the infrastructure level.
 ## retry_storm (`gateway`)
 **Mechanism:** `services/gateway/orchestrator.py::_score_with_retry`. On any
 non-2xx from `fraud`, retries immediately up to 3 more times -- no backoff,
 no jitter, no circuit breaker.
 **Symptom:** when `sanctions` is also failing or slow, the number of
 `sanctions` "lookup" log lines per transaction multiplies (up to 4x), and
 downstream load spikes exactly when the downstream is already unhealthy --
 the opposite of what you want.
 ## no_batching (`gateway`)
 **Mechanism:** `services/gateway/orchestrator.py::create_transaction`. Splits
 the transaction into 1-5 synthetic "line items" and calls `fraud` once per
 item, sequentially, instead of once for the whole transaction.
 **Symptom:** `sanctions` lookups per transaction multiply (roughly 2-5x) and
 end-to-end latency for `POST /api/transactions` grows proportionally, even
 though sanctions/fraud themselves are healthy.
 ## unbounded_cache (`insights`)
 **Mechanism:** `services/insights/chaos.py` / `app.py`. Instead of the normal
 Redis cache (keyed by account + current chaos-flag fingerprint, TTL below),
 memoizes summaries in a module-level dict keyed by `(account_id, fingerprint,
 time.time_ns())` -- a key that is different on every single call. It never
 hits, and it never evicts.
 **Symptom:** `insights`' resident memory grows monotonically under repeated
 identical-looking requests and never plateaus or drops; cache hit rate (if you
 were watching Redis) would be zero -- the "cache" provides no speedup, only
 a leak.
 ## slow_consumer (`notifier`)
 **Mechanism:** `services/notifier/consumer.py`. Sleeps 250ms per stream
 message before acking, inside the `notifier` consumer-group loop.
 **Symptom:** the `notifier` consumer-group lag on `stream:transactions` grows
 under sustained load and drains slowly afterward, while the `audit` group
 (same stream, independent group) keeps up fine -- isolating the problem to
 one specific consumer, not the stream or producer.
 ## cold_cache (`insights`)
 **Mechanism:** `services/insights/app.py`. The Redis summary cache TTL drops
 from 900s to 60s.
 **Symptom:** repeated identical requests for the same account show periodic
 latency spikes roughly every 60s (a full recompute) instead of one initial
 spike followed by uniformly fast cache hits.
 ## poison_data (`ledger`)
 **Mechanism:** `services/ledger/app.py` / `repository.py::find_null_currency_row`.
 Every `POST /internal/transactions` checks whether the account has any
 existing transaction with a `NULL` currency (one of the seeded malformed row
 varieties, §8). When the flag is on, that row's `currency` is accessed with no
 null guard (`.upper()` on `None`), which raises and surfaces as an unhandled
 HTTP 500. When off, the same check is guarded and does nothing.
 **Symptom:** a specific, small set of accounts reliably 500 on every
 transaction attempt while the flag is on, and behave completely normally
 while it's off -- input data that was always there, suddenly fatal.
 ## chatty_health (`gateway`)
 **Mechanism:** `services/gateway/orchestrator.py::_run_chatty_health_probe`.
 Before doing any real work, `POST /api/transactions` calls `GET /healthz`
 sequentially on all five downstreams, tagged with an `X-Ledgerline-Probe`
 header so the extra hits are distinguishable from routine health polling.
 **Symptom:** every transaction request produces five extra, sequential
 health-check round trips before the real work even starts, inflating latency
 by a fixed per-request tax that has nothing to do with load or data size.
@@ -0,0 +1,373 @@
 # Phase 0 — Technical Spec: "Ledgerline" base project
 > **How to use this file:** paste the whole thing into your coding agent as the task
 > spec. It is written to be executed without further clarification. Read
 > §2 (Non-goals) yourself before you start — the deliberate omissions are the
 > point of the exercise.
 ---
 ## 1. Mission
 Build **Ledgerline**, a deliberately imperfect microservice system that simulates a
 small payments/finance backend. It exists for one reason: to be a realistic subject
 for observability work in later phases. The business logic is fake and does not
 need to be correct, defensible, or secure. What it *does* need to be is
 **structurally realistic** — real network hops, real database queries, real async
 message passing, real contention, real latency distributions, and real failure modes.
 The system must be runnable with `make up && make seed && make smoke` on a laptop.
 ---
 ## 2. Non-goals — read carefully
 These are **hard constraints**, not preferences. Violating them destroys the value
 of the project.
 - **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client,
  no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no
  APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
 - **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add
  indexes, connection-pool tuning, batching, backoff, or caching beyond what is
  specified.
 - **DO NOT add authentication, authorization, rate limiting, or input hardening.**
  A single hardcoded demo user is fine.
 - **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the
  deliverable.
 - **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only.
 - **DO NOT abstract prematurely.** Duplication across services is acceptable and
  preferred over a clever shared framework.
 The only logging permitted is Python's standard `logging` module with a plain-text
 formatter (§9).
 ---
 ## 3. Technology
 | Concern | Choice |
 |---|---|
 | Language | Python 3.12 |
 | Web framework | FastAPI + `uvicorn` (one worker per service) |
 | HTTP client | `httpx` (async) |
 | DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` |
 | Cache / queue | Redis 7 (`redis-py` asyncio client) |
 | Async messaging | Redis **Streams** (not Celery, not RabbitMQ) |
 | Validation | Pydantic v2 |
 | Packaging | One shared `Dockerfile`, one image, `CMD` differs per service |
 | Orchestration | Docker Compose |
 | Schema | A single idempotent `db/schema.sql` — no Alembic |
 | Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework |
 Redis Streams is chosen deliberately: it has **no automatic context propagation**,
 so a later phase has to implement it manually. Do not swap it for something with
 built-in instrumentation.
 ---
 ## 4. Services
 Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`.
 | # | Service | Port | Role | Stores |
 |---|---|---|---|---|
 | 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) |
 | 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
 | 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
 | 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) |
 | 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis |
 | 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres |
 | 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |
 Every HTTP service exposes:
 - `GET /healthz` → `{"status": "ok", "service": "<name>"}` — liveness, no dependency checks
 - `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable
 - `GET /` → `{"service": "<name>", "version": "0.1.0"}`
 ---
 ## 5. Domain model (`db/schema.sql`)
 ```
 accounts            (id UUID pk, holder_name text, currency char(3),
                     balance_minor bigint, is_hot boolean default false,
                     created_at timestamptz)
 merchant_categories (id smallint pk, code text unique, label text,
                     risk_weight numeric(3,2))
 transactions        (id UUID pk, account_id UUID, kind text,          -- deposit|withdrawal|purchase
                     amount_minor bigint, currency char(3),
                     merchant_name text, category_id smallint,
                     status text,                                     -- pending|settled|rejected
                     risk_score numeric(4,3), created_at timestamptz)
 sanctions_entries   (id serial pk, normalized_name text, country char(2),
                     list_source text, added_at timestamptz)
 notifications       (id UUID pk, transaction_id UUID, channel text,   -- email|push|sms
                     status text, attempts int, sent_at timestamptz)
 audit_events        (id bigserial pk, transaction_id UUID, event_type text,
                     payload jsonb, recorded_at timestamptz)
 ```
 **Index policy — important.** Create only these indexes:
 - primary keys (implicit)
 - `accounts(holder_name)`
 - `sanctions_entries(normalized_name)`
 Deliberately **omit** `transactions(account_id, created_at)` and
 `transactions(category_id)`. Put the `CREATE INDEX` statements for those in
 `db/optional_indexes.sql`, applied only by `make fix-index`.
 ---
 ## 6. Request flows
 ### 6.1 `POST /api/transactions` — the primary flow
 The trace this produces should be ~4 levels deep with one async fork.
 ```
 browser
  └─ gateway  POST /api/transactions
       ├─ ledger   POST /internal/transactions          (create as `pending`, row lock on account)
       ├─ fraud    POST /internal/score                 (risk scoring)
       │     └─ sanctions  POST /internal/lookup        (watchlist check — slow vendor)
       ├─ ledger   POST /internal/transactions/{id}/settle   (or /reject if risk_score > 0.8)
       ├─ redis    XADD  stream:transactions            (fire-and-forget)
       │              ├─ notifier  consumer group `notifier`
       │              └─ audit     consumer group `audit`
       └─ insights GET  /internal/accounts/{id}/summary (cached)
 ```
 Response body:
 ```json
 {
  "transaction_id": "…", "status": "settled", "risk_score": 0.12,
  "balance_minor": 481200, "summary": { "…": "…" },
  "request_id": "…", "elapsed_ms": 843
 }
 ```
 ### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow
 `gateway → insights`. `insights` runs a 120-day aggregation over `transactions`.
 This is where the N+1 and the missing index bite.
 ### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown
 ### 6.4 Admin
 - `GET  /api/admin/chaos` → current flag values
 - `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}`
 - `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}`
 Flags live in a Redis hash `chaos:flags` so every service reads the same state
 without a restart. Each service reads the hash on **every request** (a tiny bit of
 extra Redis load is fine and realistic).
 ---
 ## 7. The pathology catalogue
 Each pathology lives in a clearly named `chaos.py` module inside its owning service,
 with a docstring naming it. All flags default **off**.
 | Flag | Service | Behaviour when enabled |
 |---|---|---|
 | `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). |
 | `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. |
 | `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. |
 | `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) |
 | `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. |
 | `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. |
 | `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. |
 | `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. |
 | `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. |
 | `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. |
 | `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. |
 | `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. |
 | `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. |
 **Profiles**
 - `clean` — all off.
 - `light` — `slow_sanctions`, `cold_cache`.
 - `nightmare` — all thirteen on.
 - `random` — pick a random subset of size 3–6, using a seed derived from
  `time.time()` and **not logged anywhere**. Store the active set in Redis under
  `chaos:flags` only, never write it to a file or print it. (This makes the later
  bottleneck-hunting exercise a genuine hunt.)
 Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the
 symptom a reader should expect. Do **not** reproduce or summarise its contents in
 your final chat response.
 ---
 ## 8. Seed data (`scripts/seed.py`)
 Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert.
 - **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD,
  10% GBP. Balances log-normal, median ≈ 250 000 minor units.
  Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets.
 - **45 merchant categories** with `risk_weight` between 0.01 and 0.95.
 - **250 000 transactions** spread over the last 120 days.
  - Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
  - Amounts log-normal.
  - Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest
    on a Zipf-ish distribution.
  - `status` 92% `settled`, 5% `rejected`, 3% `pending`.
 - **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account
  holders, so a meaningful fraction of requests genuinely score as high risk.
 - **~2% malformed rows** distributed through `transactions`:
  - `currency` = `NULL`
  - negative `amount_minor` on a `deposit`
  - `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence
  - `created_at` set 3 days in the future
  - `category_id` pointing at a non-existent category
 Print a summary table at the end (row counts per table, hot account IDs, count of
 malformed rows). Seeding 250k rows must complete in under 60 seconds — use
 `COPY` / `executemany`, not per-row inserts.
 ---
 ## 9. Logging (the deliberately bad baseline)
 Every service configures:
 ```python
 logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
 )
 ```
 - Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value.
 - `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it
  as a header on every downstream `httpx` call. Each service logs it manually at the
  start of each handler. **This is the only correlation mechanism that exists.**
  It is intentionally weak — it does not survive the Redis Stream hop, it carries no
  timing, and it cannot express parent/child relationships.
 - Set `PYTHONUNBUFFERED=1` in the Dockerfile.
 - Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK.
 ---
 ## 10. Frontend
 Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`,
 `app.js`, `style.css`. No bundler, no npm, no CDN dependencies.
 Layout, top to bottom:
 1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`),
   amount input, merchant name input, kind radio (deposit / withdrawal / purchase),
   Submit button.
 2. **Result panel** — pretty-printed JSON response, plus the browser-measured
   round-trip time in ms, plus the `request_id` shown in a monospace font that is
   easy to copy.
 3. **Load burst** — a button that fires *N* concurrent requests (N selectable:
   10 / 50 / 200) against random accounts, then renders a small table:
   count, success, errors, min / mean / p50 / p95 / max ms, measured client-side.
   This is a crude stand-in for a real load tool; a later phase replaces it with k6.
 4. **Chaos panel** — a checkbox per flag reflecting current state from
   `GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile
   buttons (`clean`, `light`, `nightmare`, `random`).
 5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s
   via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green
   or red dot per service.
 Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.
 ---
 ## 11. Repository layout
 ```
 ledgerline/
 ├─ Makefile
 ├─ docker-compose.yml
 ├─ Dockerfile                  # single image for all Python services
 ├─ pyproject.toml              # or requirements.txt — one dependency set
 ├─ .env.example
 ├─ README.md
 ├─ db/
 │  ├─ schema.sql
 │  └─ optional_indexes.sql
 ├─ scripts/
 │  ├─ seed.py
 │  └─ smoke.py
 ├─ common/
 │  ├─ __init__.py
 │  ├─ settings.py              # pydantic-settings, env-driven
 │  ├─ db.py                    # async engine/session factory
 │  ├─ redis_client.py
 │  ├─ http.py                  # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
 │  └─ logging_config.py        # the plain-text logging setup from §9
 ├─ services/
 │  ├─ gateway/    (app.py, chaos.py, orchestrator.py, static/)
 │  ├─ ledger/     (app.py, chaos.py, repository.py)
 │  ├─ fraud/      (app.py, chaos.py, scoring.py)
 │  ├─ sanctions/  (app.py, chaos.py, repository.py)
 │  ├─ insights/   (app.py, chaos.py, analytics.py)
 │  ├─ notifier/   (app.py, chaos.py, consumer.py)
 │  └─ audit/      (worker.py, consumer.py)
 └─ docs/
   └─ PATHOLOGIES.md
 ```
 `docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use
 `depends_on` with `condition: service_healthy` for postgres and redis. Mount the
 source read-write and run uvicorn with `--reload` so iteration is fast.
 Reserve — but leave empty and unreferenced — a top-level `observability/` directory
 with a `.gitkeep`. Later phases fill it.
 ---
 ## 12. Makefile targets
 | Target | Effect |
 |---|---|
 | `make up` / `make down` | compose up -d / down |
 | `make logs` | `docker compose logs -f` |
 | `make seed` | apply `db/schema.sql`, run `scripts/seed.py` |
 | `make reset` | `down -v`, `up`, `seed` |
 | `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s |
 | `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile |
 | `make fix-index` | apply `db/optional_indexes.sql` |
 | `make drop-index` | drop them again |
 ---
 ## 13. Acceptance criteria
 The task is done when all of these hold on a clean checkout:
 1. `make reset` completes in under 3 minutes and prints the seed summary table.
 2. `make smoke` passes with `chaos-clean`.
 3. The frontend loads at `http://localhost:8000`, the account dropdown is populated,
   and a submitted transaction returns a settled or rejected result.
 4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under
   ~250 ms and zero errors.
 5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error
   rate between 2% and 10% — degraded but not fully broken.
 6. Toggling any single flag via the chaos panel changes behaviour **without a
   restart**.
 7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`.
 8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns
   nothing.
 9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies.
 In your final message, report only: services started, row counts seeded, the
 p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you
 deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.
@@ -0,0 +1,508 @@
 # Phase 0a — Technical Spec: Ledgerline integration test suite
 > **How to use this file:** paste it into the **test-authoring agent** together
 > with `phase-0-base-project-spec.md`. That agent writes only `tests/`.
 > A second agent then implements the services against the resulting suite.
 > See §1 for the workflow contract between them.
 ---
 ## 1. The two-agent workflow
 ### Agent A — test author (this spec)
 - Reads `phase-0-base-project-spec.md` (the system contract) and this file.
 - Creates **only** these paths:
  ```
  tests/**            pytest.ini (or pyproject [tool.pytest])
  requirements-test.txt
  tests/README.md
  tests/RED-BASELINE.md
  ```
  Plus the `test-*` targets appended to the `Makefile`.
 - Creates **nothing** under `services/`, `common/`, `db/`, `scripts/`,
  `observability/` or `loadtest/`. Not even a stub. If a test needs a fixture
  from the system, it asserts against the contract in §4 rather than importing
  application code.
 - **Imports nothing from the application.** Tests talk to the system over HTTP,
  Postgres, Redis, and the Docker CLI. No `from services.gateway import …`.
  This is what keeps them behavioural rather than implementation-coupled.
 - Runs the suite against an empty repo and confirms it is **red**, then writes
  `tests/RED-BASELINE.md` listing, per test, *why* it currently fails
  (connection refused / 404 / assertion / missing table). A test that passes
  against nothing is a broken test — find it and fix it.
 ### Agent B — implementer
 - Reads `phase-0-base-project-spec.md` and treats `tests/` as a **read-only
  contract**.
 - **Must not modify, delete, skip, xfail, or loosen any test**, or change
  `pytest.ini`, markers, thresholds, or fixtures. If a test appears wrong,
  impossible, or contradictory: **stop and report it**, don't edit it.
 - Must implement each pathology by the mechanism described in
  `phase-0-base-project-spec.md` §7 — not as a generic `sleep()` tuned to make a
  timing assertion pass. Several assertions in §6 are deliberately built so that
  faking is harder than implementing (query counts, `EXPLAIN` plan shapes, Redis
  consumer lag, process RSS). Treat that as the intent everywhere, including
  where the assertion happens to be simpler.
 - Done when `make test` is green three consecutive runs.
 ---
 ## 2. Design principles for this suite
 These are the rules the tests must obey. They matter more than any individual
 test.
 1. **Assert goals, not implementation.** No test may reference a function name,
   module path, class, SQL string, log format, or internal attribute. A test may
   reference the *public contract*: HTTP routes, status codes, JSON field names,
   table names, Redis stream and consumer-group names, chaos flag names.
 2. **Prefer assertions that are hard to satisfy without the real mechanism.**
   Wall-clock latency alone is fakeable with `sleep()`. Query counts from
   `pg_stat_statements`, `EXPLAIN` plan node types, Redis consumer-group lag,
   container RSS, and log-line counts on a *different service* are not. Where
   both are available, assert on both.
 3. **Measure differentially, never absolutely.** With one exception (§5.7),
   every pathology test compares flag-on against flag-off in the same run, on
   the same machine, and asserts a *ratio*. Absolute millisecond thresholds do
   not survive a different laptop.
 4. **Design for a noisy laptop.** Generous margins, warm-up requests discarded,
   median-of-N measurement runs. A test that fails when Slack is open is worse
   than no test.
 5. **Fast tests and slow tests are separate.** `make test-contract` must run in
   under 60 seconds. Pathology tests may take minutes and live behind their own
   marker.
 6. **Every test restores state.** A test that enables a chaos flag must reset
   the profile in teardown, even on failure.
 ---
 ## 3. Stack, layout, and execution
 Tests run **on the host**, against the Compose stack on `localhost`. They do not
 run inside a container and do not use testcontainers — the stack is started by
 `make up` beforehand.
 ```
 tests/
 ├─ README.md
 ├─ RED-BASELINE.md
 ├─ conftest.py                # fixtures: clients, chaos control, seed manifest
 ├─ harness/
 │  ├─ load.py                 # concurrent request driver + LatencyReport
 │  ├─ pg.py                   # pg_stat_statements deltas, EXPLAIN helpers
 │  ├─ streams.py              # Redis XINFO GROUPS helpers
 │  └─ containers.py           # docker stats / docker logs helpers
 ├─ test_contract_health.py
 ├─ test_contract_transactions.py
 ├─ test_contract_insights.py
 ├─ test_contract_async.py
 ├─ test_contract_chaos.py
 ├─ test_contract_frontend.py
 ├─ test_seed_data.py
 ├─ test_baseline_health.py
 ├─ test_pathologies.py
 ├─ test_nightmare.py
 └─ test_no_observability.py
 ```
 Dependencies (`requirements-test.txt`): `pytest`, `pytest-asyncio`,
 `pytest-timeout`, `httpx`, `psycopg[binary]`, `redis`. Docker access via
 `subprocess` calling the `docker` CLI — no SDK.
 ### Markers
 | Marker | Runtime | Contents |
 |---|---|---|
 | `contract` | < 60 s | Functional behaviour, chaos-clean |
 | `seed` | < 20 s | Seed data integrity |
 | `baseline` | < 90 s | The clean system is genuinely healthy |
 | `pathology` | 5–12 min | One test per pathology, differential |
 | `slow` | — | Additional marker on the soak-ish ones (`unbounded_cache`, `cold_cache`) |
 | `guard` | < 5 s | The "no observability yet" assertions |
 Configure `pytest.ini` with `--strict-markers`, `asyncio_mode = auto`, and a
 global `timeout = 300`.
 ### Make targets to append
 | Target | Command |
 |---|---|
 | `test` | everything except `slow` |
 | `test-contract` | `-m "contract or seed or guard"` |
 | `test-baseline` | `-m baseline` |
 | `test-pathology` | `-m "pathology and not slow"` |
 | `test-slow` | `-m slow` |
 | `test-all` | no marker filter |
 ---
 ## 4. Contract additions required for testability
 These extend `phase-0-base-project-spec.md`. Agent B must implement them; Agent A
 may assume them.
 ### 4.1 Seed manifest
 `scripts/seed.py` writes `db/seed_manifest.json` **and** a `seed_manifest` table
 (single row, `data jsonb`) with the same content. Tests read the table, so they
 don't depend on a host file path.
 ```json
 {
  "seed": 42,
  "generated_at": "…",
  "counts": { "accounts": 800, "transactions": 250000,
              "merchant_categories": 45, "sanctions_entries": 12000,
              "malformed_transactions": 5013 },
  "hot_account_ids": ["…", "…", "…", "…", "…"],
  "cold_account_ids_sample": ["…"],            // 50 non-hot accounts, for spread tests
  "sanctioned_account_ids": ["…"],             // accounts whose holder matches the watchlist
  "poison_account_ids": ["…"],                 // accounts owning malformed rows
  "clean_account_id": "…"                      // no malformed rows, not hot, not sanctioned
 }
 ```
 ### 4.2 Response envelope
 `POST /api/transactions` → **201** on success:
 ```json
 { "transaction_id": "<uuid>", "status": "settled|rejected",
  "risk_score": 0.0-1.0, "balance_minor": 0,
  "summary": { }, "request_id": "<uuid>", "elapsed_ms": 0 }
 ```
 Errors return a consistent envelope with the appropriate status:
 ```json
 { "error": "<machine_readable_code>", "detail": "<human string>",
  "request_id": "<uuid>" }
 ```
 Status mapping: `400` invalid input, `404` unknown account, `422` validation,
 `502` downstream returned an error, `504` downstream timed out, `500`
 unhandled — which is exactly what `poison_data` produces, and what §6.12 asserts.
 ### 4.3 Chaos API
 - `GET /api/admin/chaos` → `{"profile": "<name|custom>", "flags": {"<flag>": bool, …}}`
  containing **all thirteen** flags, always.
 - `POST /api/admin/chaos` → `{"flag": "<name>", "enabled": bool}`; returns the
  full new state. Unknown flag → `400`.
 - `POST /api/admin/chaos/profile` → `{"profile": "clean|light|nightmare|random"}`;
  returns the full new state. For `random`, the response reports the flags —
  **but `make chaos-random` must not print them**, and the seed is never logged.
 - `GET /api/health/all` → `{"services": {"<name>": "ok"|"unhealthy", …}}` for all
  six HTTP services.
 ### 4.4 `pg_stat_statements`
 The Postgres image must load `pg_stat_statements` (`shared_preload_libraries`)
 and the extension must be created by `db/schema.sql`. Tests use
 `pg_stat_statements_reset()` around a single request to count queries.
 ### 4.5 Deterministic seeding
 `SEED` env var, default `42`. Same seed ⇒ identical manifest, identical row
 counts, identical hot/poison account IDs.
 ---
 ## 5. Goal-to-test traceability
 Every test must map to one of these goals. Put the goal ID in the test's
 docstring. This is what keeps the suite about intent rather than mechanics.
 | ID | Goal | Where |
 |---|---|---|
 | **G1** | A transaction traverses all seven services and returns a coherent result | §5.1 |
 | **G2** | The system is genuinely multi-service, not a collapsed monolith | §5.2 |
 | **G3** | Async work happens after the response, via two independent consumer groups | §5.3 |
 | **G4** | The seed data is large, skewed, and contains the intended poison rows | §5.4 |
 | **G5** | Chaos flags toggle at runtime without a restart | §5.5 |
 | **G6** | Each pathology measurably degrades the system when enabled | §6 |
 | **G7** | The clean baseline is actually healthy — otherwise G6 means nothing | §5.6 |
 | **G8** | Nightmare mode degrades but does not collapse | §5.7 |
 | **G9** | Failure is contained: timeouts fire, the gateway doesn't wedge | §5.8 |
 | **G10** | No observability tooling is present yet | §5.9 |
 | **G11** | The frontend and its supporting endpoints are served | §5.10 |
 ### 5.1 G1 — end-to-end transaction (`contract`)
 - Submitting a valid transaction for `clean_account_id` returns 201 with the full
  envelope; `transaction_id` is a UUID; `status` ∈ {settled, rejected};
  `risk_score` ∈ [0,1]; `elapsed_ms` > 0.
 - The transaction is durable: a row with that ID exists in `transactions` with a
  terminal status within 2 s.
 - The balance moves in the correct direction for a deposit and for a withdrawal,
  by exactly `amount_minor`, when the transaction settles.
 - A transaction for a `sanctioned_account_ids` holder yields a materially higher
  `risk_score` than one for `clean_account_id` (assert the ordering, not a value).
 - `GET /api/accounts/{id}/insights` returns a summary for a seeded account.
 - Unknown account → 404 with the error envelope. Negative amount on a deposit →
  400 or 422. Missing required field → 422.
 ### 5.2 G2 — really seven services (`contract`)
 - All six HTTP services answer `/healthz` on their own port with their own name.
 - `/readyz` returns 200 for all six.
 - `GET /api/health/all` reports all six as ok.
 - Stopping `sanctions` (`docker compose stop sanctions`) causes
  `GET /api/health/all` to report it unhealthy within 15 s, and
  `POST /api/transactions` to fail with 502 or 504 — **not** 200. Restart it in
  teardown and wait for ready.
  *(This is the test that catches a collapsed monolith: if transactions still
  succeed with `sanctions` stopped, the service isn't really in the path.)*
 ### 5.3 G3 — asynchronous work (`contract`)
 - After a successful transaction, a row appears in `notifications` **and** in
  `audit_events` for that `transaction_id` within 10 s.
 - Both appear — proving two independent consumer groups, not one consumer doing
  both jobs.
 - `XINFO GROUPS stream:transactions` reports exactly two groups, named
  `notifier` and `audit`.
 - The API response returns **before** the notification row exists at least once
  across 20 attempts, proving the work is genuinely off the request path.
  *(Sample the DB immediately on response; assert at least one miss. Tolerant of
  a fast machine winning the race sometimes — hence 20 attempts, not 1.)*
 ### 5.4 G4 — seed data (`seed`)
 Against the manifest and the database:
 - Row counts match the manifest and the spec's targets within ±2%.
 - Exactly 5 accounts have `is_hot = true`, and they match `hot_account_ids`.
 - The 5 hot accounts own between 20% and 40% of all transactions.
 - Transactions span ≥ 110 distinct days.
 - Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each).
 - Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points).
 - Malformed rows exist in all five described varieties: null currency, negative
  amount on a deposit, non-ASCII merchant name, future `created_at`, orphan
  `category_id`. Assert **at least one of each**, and 1%–4% overall.
 - ≥ 30 sanctions entries match seeded holder names.
 - Re-running the seed with the same `SEED` yields an identical manifest
  (compare the `counts` and `hot_account_ids`).
 - Neither `transactions(account_id, created_at)` nor `transactions(category_id)`
  exists in `pg_indexes` after a fresh seed.
 ### 5.5 G5 — chaos control (`contract`)
 - `GET /api/admin/chaos` lists all thirteen flags by name.
 - Toggling a single flag changes only that flag.
 - Each profile sets the expected shape: `clean` → all false; `nightmare` → all
  true; `light` → exactly the two documented flags.
 - **No restart:** with `chaos-clean` set, record the `insights` latency; enable
  `n_plus_one` via the API; without restarting any container, the query count for
  the same request rises. Assert on the query-count delta (§6.3's helper), so
  this test proves live reconfiguration rather than timing noise.
 - Unknown flag name → 400.
 - Teardown restores `clean`.
 ### 5.6 G7 — the clean baseline is healthy (`baseline`)
 With `chaos-clean`, after 20 discarded warm-up requests, drive 200 requests at
 concurrency 10 against random cold accounts:
 - error rate == 0
 - p95 < 400 ms *(the one absolute threshold in the functional set; if it fails on
  a slow machine, that machine cannot run the pathology suite meaningfully
  either, and the failure message must say so)*
 - p99 / p50 < 5 — the clean system has no fat tail
 - `insights` for a cold account issues fewer than 10 SQL statements
 - Redis consumer lag returns to 0 within 15 s of the burst finishing
 ### 5.7 G8 — nightmare degrades but survives (`pathology`)
 Same driver, `chaos-clean` then `chaos-nightmare`, 300 requests at concurrency 20
 each:
 - nightmare p95 ≥ **5×** clean p95
 - nightmare error rate between **2% and 15%** — degraded, not dead
 - at least 80% of requests still return a valid envelope (either success or a
  structured error, never a connection reset or an empty body)
 - `GET /api/health/all` still responds within 5 s during the burst
 ### 5.8 G9 — failure containment (`contract`)
 - With `sanctions_errors` at its maximum and `retry_storm` off, no request takes
  longer than 10 s. The 3 s downstream timeout must actually fire.
 - With `sanctions` stopped entirely, `POST /api/transactions` returns 502/504
  within 10 s, and the gateway still answers `/healthz` in under 1 s throughout.
 - No request ever returns a 200 with a `null` `transaction_id`.
 ### 5.9 G10 — no observability yet (`guard`)
 Static and runtime assertions that the Phase 0 "before" state is intact:
 - No file under `services/` or `common/` imports `opentelemetry`,
  `prometheus_client`, `structlog`, `sentry_sdk`, or `elasticapm`.
 - Neither `requirements*.txt` nor `pyproject.toml` lists them (excluding
  `requirements-test.txt`).
 - No service exposes `/metrics` (assert 404 on all six).
 - `docker compose logs gateway` emits plain text, not JSON: sample 20 lines and
  assert none parse as a JSON object.
 > This test class is **expected to be deleted at the start of Phase 2**, not
 > maintained. Say so in its module docstring.
 ### 5.10 G11 — frontend (`contract`)
 - `GET /` returns 200, `text/html`, and references `app.js` and `style.css`.
 - Those two assets return 200 with sensible content types.
 - `GET /api/accounts` returns ≥ 100 accounts, each with `id`, `holder_name`,
  `currency`, `balance_minor`.
 No browser automation. Serving and shape only.
 ---
 ## 6. G6 — pathology tests
 **One test per flag.** Each follows the same skeleton:
 ```
 1. set chaos-clean
 2. warm up (discard)
 3. measure  →  baseline
 4. enable exactly one flag via the API
 5. warm up (discard)
 6. measure  →  degraded
 7. assert the ratio, with the failure message printing both measurements
 8. teardown: chaos-clean
 ```
 Measurement runs the median of 3 for latency-based assertions. Failure messages
 must always include both raw numbers — a bare `assert ratio > 3` is useless at
 2 a.m.
 | § | Flag | Primary signal (hard to fake) | Secondary signal | Assertion |
 |---|---|---|---|---|
 | 6.1 | `slow_sanctions` | — | p99/p50 of `POST /api/transactions` | degraded p99/p50 ≥ **6**, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99 |
 | 6.2 | `sanctions_errors` | count of 502/504 responses over 300 requests | — | degraded error rate ∈ [1%, 10%]; baseline == 0 |
 | 6.3 | `n_plus_one` | `pg_stat_statements` call delta across **one** insights request | latency ratio | degraded ≥ **50** statements; baseline ≤ 10; latency ratio ≥ 2 |
 | 6.4 | *missing index* (`make drop-index` / `fix-index`, not a chaos flag) | `EXPLAIN (ANALYZE, FORMAT JSON)` on the insights aggregation contains a `Seq Scan` on `transactions`; after `fix-index` it contains an `Index Scan`/`Bitmap Index Scan` | latency ratio | plan node types as stated; unindexed latency ≥ **3×** indexed. Teardown restores the dropped state. |
 | 6.5 | `pool_starvation` | throughput (req/s) at concurrency 5 vs 25 | p95 ratio | with the flag on, throughput at c=25 is ≤ **1.4×** throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25 |
 | 6.6 | `row_lock_contention` | 25 concurrent transactions against **one hot account** vs 25 spread across 25 cold accounts, both with the flag on | — | single-hot-account p95 ≥ **3×** spread p95; with the flag off the same comparison is ≤ 1.8× |
 | 6.7 | `blocking_event_loop` | wall-clock for 10 concurrent requests ÷ wall-clock for 1 request | — | degraded ratio ≥ **6** (serialised); baseline ratio ≤ 2.5 (concurrent) |
 | 6.8 | `retry_storm` | count of `sanctions` container log lines matching a lookup, per transaction, with `sanctions_errors` also on | — | degraded ≥ **2.0** lookups/transaction; baseline (retry off, errors on) ≤ 1.2. *Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.* |
 | 6.9 | `no_batching` | `sanctions` lookup log lines per transaction, `sanctions_errors` **off** | latency ratio | degraded ≥ **2.0** lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5 |
 | 6.10 | `unbounded_cache` (`slow`) | `insights` container RSS via `docker stats` before vs after 2000 requests with **distinct** query parameters, sampled 30 s after the burst ends | — | degraded RSS growth ≥ **20 MB** and still elevated at +30 s; baseline growth ≤ 8 MB |
 | 6.11 | `slow_consumer` | `XINFO GROUPS stream:transactions` → `lag`/`pending` for the `notifier` group after a 150-transaction burst | drain time | degraded lag ≥ **30** at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s |
 | 6.12 | `cold_cache` (`slow`) | same insights request at 1 rps for 150 s; count samples > 3× the median | — | degraded ≥ **2** spikes (a 60 s TTL over 150 s); baseline ≤ 1 |
 | 6.13 | `poison_data` | request against each `poison_account_ids` entry | — | degraded produces ≥ 1 response with status **500**; baseline produces zero 500s across the same requests (4xx or 200 only) |
 | 6.14 | `chatty_health` | `/healthz` hits on downstream containers during a single transaction, counted from logs | — | degraded ≥ **5** healthz lines across downstreams per transaction; baseline == 0 |
 **Note on log-line counting (6.8, 6.9, 6.14).** At Phase 0 the plain-text log is
 the only cross-service observability surface that exists — which is precisely the
 deficiency this whole project sets out to fix. Implement these helpers behind
 `harness/containers.py::count_log_lines(service, pattern, since)` so there is
 exactly one place to change. **From Phase 3 these assertions should be rewritten
 against structured logs, and from Phase 4 against span counts** — at which point
 they become both simpler and more robust. Note this in `tests/README.md`; it is a
 deliberate teaching artefact, not a shortcut.
 **Anti-fake note for Agent B.** 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied
 by inserting a `sleep()`: they require, respectively, real repeated queries, a
 real missing index, real unbounded allocation, real consumer slowness, and a real
 unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12)
 are stated as ratios for the same reason — passing them by faking is more work
 than implementing the mechanism described in the Phase 0 spec.
 ---
 ## 7. The measurement harness
 `harness/load.py` provides one primitive:
 ```python
 async def drive(
    method: str, url_factory: Callable[[], tuple[str, dict]],
    total: int, concurrency: int,
    warmup: int = 20, timeout: float = 15.0,
 ) -> LatencyReport
 ```
 `LatencyReport` carries `count, ok, errors, error_rate, status_counts,
 p50, p95, p99, max, throughput_rps, wall_seconds`, plus a `__str__` suitable for
 dropping straight into an assertion message.
 Rules:
 - Warm-up requests are issued and discarded before timing starts.
 - Percentiles from the raw sorted sample, nearest-rank. No interpolation, no
  averaging of percentiles across runs — **median of the p95s**, never the mean.
 - A closed-loop driver (fixed concurrency) is used deliberately for its
  simplicity here. Note in `tests/README.md` that this under-measures latency
  under saturation (coordinated omission) and that Phase 6 replaces it with k6's
  arrival-rate executors. Don't try to fix it in the test suite.
 - `measure_median(n=3, fn)` helper for the pathology tests.
 `harness/pg.py`:
 ```python
@contextmanager
 def statement_count(dsn) -> Counter   # resets pg_stat_statements, yields, returns delta
 def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON)
 def plan_node_types(plan: dict) -> set[str]
 ```
 `harness/streams.py`: `group_lag(group)`, `wait_for_drain(group, timeout)`.
 `harness/containers.py`: `rss_mb(service)`, `count_log_lines(service, pattern,
 since)`, `stop(service)`, `start(service)`, `wait_ready(service, timeout)`.
 ---
 ## 8. Fixtures (`conftest.py`)
 | Fixture | Scope | Purpose |
 |---|---|---|
 | `stack_up` | session | Assert the Compose stack is running and all six services are ready; **fail fast with a clear message** ("run `make up && make seed` first") rather than timing out |
 | `seeded` | session | Assert the seed manifest exists and counts are non-zero; fail fast otherwise |
 | `manifest` | session | Parsed seed manifest |
 | `api` | function | `httpx.AsyncClient` against the gateway, 15 s timeout |
 | `dsn` | session | Postgres DSN from env |
 | `redis_client` | session | — |
 | `chaos` | function | `chaos.set_profile(name)`, `chaos.set_flag(name, bool)`, `chaos.get()`. **Autouse teardown resets to `clean`**, even on failure |
 | `fresh_balances` | function | Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs |
 Every environment value (ports, DSN, Redis URL) comes from env with the Compose
 defaults as fallbacks. No hardcoded values buried in test bodies.
 ---
 ## 9. `tests/README.md` — what Agent A must write
 - How to run each marker set, and the prerequisite (`make up && make seed`).
 - The measurement methodology and why ratios are used rather than absolutes.
 - The known-flaky list: 6.10 and 6.12 are the least stable; document the retry
  strategy and what a genuine failure looks like versus noise.
 - The coordinated-omission caveat on the closed-loop driver.
 - The Phase 3/4 migration note for the log-counting helpers.
 - An explicit statement that `test_no_observability.py` is deleted at Phase 2.
 ---
 ## 10. Acceptance criteria
 For **Agent A**:
 1. `pytest --collect-only` succeeds and every test carries exactly one goal ID in
   its docstring and at least one marker.
 2. `grep -rn "from services\|from common\|import services\|import common" tests/`
   returns nothing.
 3. Every §5 and §6 item has a corresponding test; `tests/README.md` contains the
   goal-to-test table.
 4. The suite is **red** against an empty repo, and `tests/RED-BASELINE.md`
   records the failure reason per test. No test passes vacuously.
 5. `make test-contract` completes (failing) in under 60 s.
 For **Agent B**:
 1. `make test` green three consecutive runs.
 2. `make test-slow` green (may be run once).
 3. `git diff --stat tests/` against Agent A's commit is empty.
 4. `docs/PATHOLOGIES.md` written, and not summarised in the final response.
@@ -0,0 +1,377 @@
 # Observability & APM — Delivery Roadmap for Babak's Task
 *Part of `docs/roadmap/` — platform background and reading guidance shared by
 every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`;
 [`summary.md`](summary.md) gives the one-page overview and status of all of
 them.*
 Ten phases. Each one: concepts first, then the build, then a checkable exit
 condition. Total time-boxed at roughly **4–6 weeks part-time** (evenings +
 weekends). Nothing here is theory for theory's sake — every concept block is
 followed immediately by the place it lands in the code.
 ---
 ## Read this first — the four questions blocking you
 ### "What *is* an APM? Do you install it, use it, implement it, or build it?"
 APM (Application Performance Monitoring) is a **product category and a capability**,
 not a protocol or a single tool. A system has APM when you can answer, for
 production traffic: what's the request rate, error rate and latency distribution
 per endpoint; which services call which; where did *this specific* request spend its
 time; which database query or external call was slow; and what error was thrown,
 with the stack.
 The verb depends on the vendor model:
 - **Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM).** You
  *install* an agent that attaches to your runtime and ships to their backend. Mostly
  a deployment exercise. Turnkey, expensive, proprietary.
 - **Grafana stack — what Babak asked for.** There is no product called "Grafana APM"
  that you install. You **assemble** the capability from four layers:
  | Layer | What you do | Component |
  |---|---|---|
  | Instrumentation | *Implement* — this is code you write | OpenTelemetry SDK inside your services |
  | Pipeline | *Deploy + configure* | OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution) |
  | Storage | *Deploy* | Tempo (traces), Loki (logs), Prometheus or Mimir (metrics) |
  | Query & UI | *Configure* | Grafana + provisioned datasources with cross-links |
  So: you **implement** instrumentation, you **deploy** the pipeline and stores, you
  **configure** the correlation. The "APM" is the emergent property of all four
  working together.
 The distinction that actually matters: APM data is generated *inside* the
 application, keyed to *individual requests*, and *correlated* across signals.
 Infrastructure monitoring (CPU, disk, pod restarts) is generated *outside* the
 application and aggregated. You need both; only the first is APM.
 ### "What's the difference between Logs, Metrics, Events and Traces?"
 The clean way to hold this in your head is: **an event is the atom; the other three
 are shapes you give it.**
 | | What it is | Cost shape | Answers |
 |---|---|---|---|
 | **Event** | A structured record of one thing that happened, at one instant, with arbitrary attributes attached | Grows with traffic × attribute count | "What exactly happened here?" |
 | **Log** | An event rendered for storage/search. A *structured* log is literally an event; an unstructured log is an event you threw the schema away from | Grows with traffic | "Why did it fail?" |
 | **Metric** | A pre-aggregated numeric measurement over many events, sliced by a small set of labels | Grows with *label cardinality*, ~flat in traffic | "Is it broken? How much? Since when?" |
 | **Trace** | A causally linked tree of events that each have a *duration* and a parent — a span tree spanning process boundaries | Grows with traffic × span count | "Where did the time go, across services?" |
 Three consequences worth internalising now:
 1. **Metrics are cheap in traffic but expensive in cardinality.** Doubling your
   request rate barely changes Prometheus's memory; adding `user_id` as a label
   can kill it. Logs and traces are the opposite: cheap in dimensions, expensive
   in volume. This asymmetry drives every design decision later.
 2. **Traces are events with two extra fields:** a duration and a parent pointer.
   That's the whole trick. Everything else (service maps, critical paths, span
   metrics) is derived from those two fields.
 3. In OpenTelemetry, "events" aren't a fourth top-level signal you export
   separately. They show up as **span events** (a timestamped annotation attached
   to a span — this is how `record_exception` works) and as **log records carrying
   an `event.name` attribute**; the Events API is built on the Logs data model.
   When Babak says M.E.L.T., he most likely means "logs plus the discrete
   business/domain events you care about" — check with him, but treat it as
   *structured logs and span events*, not a separate pipeline.
 ### "Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?"
 **OpenTelemetry first, without much doubt.**
 Loki, Tempo, Prometheus and Mimir are *destinations*. They're databases with query
 languages. OpenTelemetry is the **producer-side data model and wire protocol** — it's
 the only part of this that touches your Python code, and it's the part that
 determines what the destinations can possibly show you. If you learn Loki first you
 learn Loki's label model and its promtail-isms, most of which don't transfer. If you
 learn OTel first, every backend becomes "point OTLP at a different URL."
 One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
 early, in parallel with OTel metrics — because it *constrains what you're allowed to
 emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
 (Phases 4–5), not up front.
 Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
 part of the Grafana stack and you do not need them for this task. Phase 9 covers
 them as a judgment question, which is exactly how Babak framed them.
 ### Verdict on the OneUptime article you found
 **Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
 teaching resource. Read it in **Phase 4/5**, not now; it assumes you already
 understand spans, span metrics and PromQL.
 What's genuinely good in it, and rare to find in one place:
 - The `spanmetrics` and `servicegraph` **connectors** in the Collector — this is how
  you get RED metrics and a service map "for free" from traces you already emit.
  This is the single highest-leverage config in the whole article.
 - The Tempo datasource provisioning YAML with `tracesToLogsV2`, `tracesToMetrics`,
  `serviceMap` and `nodeGraph` — the click-through correlation that makes Grafana
  feel like an APM.
 - `exemplarTraceIdDestinations` on the Prometheus datasource — jump from a latency
  spike on a graph straight into the trace that caused it.
 - The tail-sampling policy block (keep all errors, keep all slow traces, sample the
  rest).
 Two caveats:
 - Its Collector config exports logs via the **`loki` exporter**. Loki now accepts
  **OTLP natively** on its own endpoint, and that is the current recommended path —
  fewer moving parts, better attribute mapping. Use the native OTLP route.
 - It's a vendor blog (OneUptime sells a competing product). The Grafana content is
  accurate, but treat the official OpenTelemetry and Grafana docs as the source of
  truth for anything spec-level.
 - **Its `spanmetrics` / `servicegraph` Collector connectors do not apply to your
  stack.** See the next section — your platform generates those metrics in Tempo
  instead. The article's PromQL is still useful, but the metric names differ.
 ---
 ## The platform Babak gave you (`observability-ansible`)
 This changes *where* your telemetry goes and *how the pipeline is configured*. It
 does not change the learning arc, the app, or the test suite. Read this before
 Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9.
 ### What the repo is
 An Ansible project that deploys **two independent observability backends** on one
 Ubuntu/Debian host via Docker Compose:
 1. **ELK + Grafana** — Filebeat → Logstash → Elasticsearch → Kibana/Grafana
 2. **Grafana stack** — Prometheus + Alertmanager + Loki + Tempo + Grafana, with
   **Grafana Alloy** as the collector and node-exporter for host metrics
 It contains **no application code and no dashboards**. It is the *platform half* of
 Babak's task, handed to you pre-built. Your half is everything that produces
 telemetry and everything that reads it.
 Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
 `playbooks/elk-grafana.yml`, idempotent, versions pinned in `group_vars/all.yml`
 (Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
 Elastic 9.1.3). Everything lands under `/opt/observability/`.
 ### The architecture
 Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
 **Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline →
 Explore & Correlate**, with three input classes feeding it.
 ```
 App metrics       ──/metrics, Prometheus exposition──┐
 Container logs    ──Docker socket, tail──▶ Alloy ──push API──▶ Loki
 Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo
                     ("spans + resource attrs")                    │
 Node Exporter :9100 ──────scrape──────┐                            │
                                      ▼                            │
                                 Prometheus ◀──remote_write────────┘
                                  (TSDB, PromQL, rules)   span metrics + exemplars
                                      │
                             firing alerts ▼
                                 Alertmanager ──route──▶ Receivers
                                  (group, dedupe)      (Slack, Email, PagerDuty,
                                      │                 webhook, SMTP)
       Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four
              Correlation key: trace_id / exemplars
 ```
 Note what the diagram says about the OTLP box: **"spans + resource attrs"**. Traces
 only. And app metrics are labelled **"/metrics endpoint, Prometheus exposition"** —
 not OTLP. The diagram and the code agree on this; it is a design decision, not an
 oversight. Plan your instrumentation around it.
 ### One place the diagram and the code disagree
 The diagram draws app metrics as `App metrics → scrape → Alloy → /metrics →
 Prometheus`. The shipped config does not do that:
 - `config.alloy.j2` contains **no `prometheus.scrape` component at all** — Alloy
  collects logs and receives traces, nothing else.
 - `prometheus.yml.j2` scrapes `prometheus`, `node-exporter`, `loki`, `tempo`,
  `alloy:12345`, `grafana`, then loops over `prometheus_extra_scrape_jobs`. So your
  app targets are scraped by **Prometheus directly**, bypassing Alloy entirely. The
  green `/metrics` arrow from Alloy to Prometheus in the diagram is Prometheus
  scraping Alloy's **own** self-monitoring endpoint, not your app's metrics
  passing through it.
 Neither is wrong; the diagram shows the intended architecture and the code
 implements a simpler one. Closing that gap — adding `prometheus.scrape` +
 `prometheus.remote_write` to Alloy so metrics really do flow through the collector —
 is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth
 mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
 exactly the skill this task is meant to build.
 ### Four things in it that will cost you a day if you don't know them
 **1. Alloy's OTLP receiver only forwards traces.** The config is literally:
 ```
 otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
 }
 ```
 There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
 your app sends are accepted and then **silently discarded** — no error, no warning,
 just nothing arriving in Grafana. This is the single most likely thing to eat an
 evening.
 The diagram confirms this is deliberate: the OTLP input is labelled "spans +
 resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
 input. So you have three options, and you should understand all three:
 - *Follow the platform's grain (simplest):* expose a Prometheus `/metrics` endpoint
  per service and register them in `prometheus_extra_scrape_jobs` in
  `group_vars/all.yml`. Prometheus scrapes your services directly. Logs continue to
  arrive via stdout scraping.
 - *Implement what the diagram draws:* add `prometheus.scrape` +
  `prometheus.remote_write` components to Alloy so app metrics flow through the
  collector, as the architecture intends. Buys you central relabeling and one place
  to change collection policy.
 - *Go all-OTLP:* add `metrics` and `logs` outputs to the receiver, plus
  `otelcol.exporter.prometheus` → `prometheus.remote_write` and
  `otelcol.exporter.loki` → `loki.write`. One protocol for all three signals — the
  architecture you'd argue for on a greenfield platform, and the furthest from what
  ships.
 Do the first to get moving, then do at least one of the others as a deliberate
 exercise and write up the trade-off. That comparison is worth more than any single
 answer.
 **2. Span metrics come from Tempo, not from the collector.** `tempo.yml.j2` enables:
 ```
 metrics_generator:
  storage:
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
 overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]
 ```
 So RED metrics, the service graph, and exemplars already exist — generated inside
 Tempo and remote-written into Prometheus (which runs with
 `--web.enable-remote-write-receiver`). **Do not add `spanmetrics`/`servicegraph`
 connectors to Alloy** — you'd be generating the same series twice under different
 names. The metric names differ from the OneUptime article's; go find the actual
 names in Prometheus before writing any PromQL. (`local-blocks` also gives you
 TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.)
 **3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
 datasource is provisioned with:
 ```
 derivedFields:
  - datasourceUid: tempo
    matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
    name: TraceID
 ```
 Your log lines must contain **lowercase** `trace_id` (or `traceid` / `trace-id`).
 This matters more than it looks: OpenTelemetry's `opentelemetry-instrumentation-logging`
 auto-injection emits **`otelTraceID`**, which does **not** match that pattern.
 So on this platform, the structlog processor emitting a lowercase `trace_id` field
 isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus
 datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram states
 this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
 in the Explore & Correlate column hangs off that one field name.
 **4. There is no dashboard provisioning at all.** The Grafana provisioning mount
 contains only `datasources/`. Every dashboard is yours to add — which means Babak's
 "create dashboards as code, destroy and recreate in seconds" requirement is entirely
 your deliverable, and the natural home for it is *inside this Ansible role*.
 ### What this changes, phase by phase
 | Phase | Impact |
 |---|---|
 | 0 / 0a | **None.** App and test suite are unaffected. |
 | 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
 | 2 | Endpoint changes to Alloy; tracing work itself is identical. |
 | 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
 | 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
 | 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
 | 6 | None. |
 | 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
 | 8 | None, plus TraceQL metrics from `local-blocks`. |
 | 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
 | 10 | Demo now includes `ansible-playbook` rebuilding the platform. |
 ### Where to run it
 The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
 ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside
 everything else will hurt. Use a separate VM (a cheap cloud instance or a local
 Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing
 at it. That also makes the deployment *real* rather than a localhost shortcut, which
 is the point of it being Ansible.
 If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker
 socket and will scrape **every** container's stdout, including Ledgerline's,
 automatically — convenient, and worth knowing before you wonder where those logs
 came from.
 ### One thing to confirm with Babak
 Is this repo a **platform you must consume as-is**, or a **starting point you may
 modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
 assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
 extend `group_vars`. If Babak intends it as a fixed platform, several
 recommendations below change — ask before Phase 3.
 ---
 ## Your resource spine
 Pick **one paid course as a backbone** and use the free material as targeted
 supplements. Both Udemy options below ship captions, which matters for your
 subtitle-to-notes workflow.
 **Backbone — choose one:**
 | Resource | Format | Length | Cost | Best for |
 |---|---|---|---|---|
 | *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. |
 | *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. |
 If budget allows, buy both — they overlap by maybe 30% and the second half of each
 is different. If only one: take the **OpenTelemetry** one, because instrumentation
 is the part you have to *write*, and Grafana's own docs are excellent for the rest.
 **Free, and worth your time:**
 | Resource | Format | Length | Use in |
 |---|---|---|---|
 | freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
 | Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. |
 | Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. |
 | **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
 | `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
 | Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. |
 | Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 |
 | **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
 | **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. |
 | Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on |
 | Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 |
 | Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. |
 | Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
 | Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 |
 | Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. |
 | SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 |
 | Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 |
 **Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on
 your k8s cluster until everything works locally — debugging instrumentation and
 debugging Kubernetes at the same time is how these projects die. Porting it to the
 cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half
 Babak has planned for you.
@@ -0,0 +1,127 @@
 # Phase 0 — Build the base project, test-first
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`intro.md`](intro.md) · Next: [`phase-1.md`](phase-1.md)*
 **Status:** Done
 **Message:** Both halves complete and verified against the live stack, not just
 collected.
 - **0a (test suite):** 81 tests across `tests/` (11 files + a 5-module
  harness), covering all eleven goals G1–G11 and all 13 named pathologies.
  Verified genuinely red against the empty repo before any implementation
  existed; `tests/RED-BASELINE.md` records that run. `tests/README.md`
  documents the invented request-contract fields, measurement methodology,
  and known-flaky tests.
 - **0b (implementation):** all seven services (`gateway`, `ledger`, `fraud`,
  `sanctions`, `insights`, `notifier`, `audit`), `db/schema.sql` +
  `optional_indexes.sql`, `scripts/seed.py` + `smoke.py`, the frontend, and
  `docker-compose.yml`. Seeded: 800 accounts, 250,000 transactions, 45
  merchant categories, 12,000 sanctions entries, in ~12s. `make smoke` and
  `make test-contract` (64/64) pass; `make test-baseline` and
  `make test-pathology` pass except for a handful of tests documented below
  as load-sensitive rather than broken.
 - **Two changes to the frozen test contract, both made only after flagging
  them and getting an explicit go-ahead — not applied unilaterally:**
  - `test_row_counts_match_manifest_and_targets` loosened the `transactions`
    row-count check from exact equality to `>=`, since other contract tests
    in the same `make test-contract` run legitimately insert rows first.
  - `test_pool_starvation_plateaus_throughput` was **removed outright**.
    Its baseline assertion (throughput scales ≥2.5x from concurrency 5→25)
    is incompatible with this same spec's own §3 mandate of one uvicorn
    worker per service — confirmed by reproducing the identical
    non-scaling behaviour on a bare `GET /healthz` with zero backend
    logic. `pool_starvation` itself is still fully implemented
    (`services/ledger/db.py`'s dual-engine pool selection); only that one
    differential test was dropped.
 - **Known-flaky, documented rather than papered over:**
  `test_clean_baseline_is_healthy`, `test_nightmare_degrades_but_survives`,
  the baseline half of `test_blocking_event_loop_serializes_concurrent_requests`,
  and `test_missing_index_forces_seq_scan` compare a fast-path measurement
  against a slow-path one under real concurrent load, and pass at roughly a
  60–80% rate on ordinary dev hardware depending on momentary host
  contention. See "Known-flaky tests" in `tests/README.md`.
 - **Notable implementation deviations** (none change observable behaviour of
  the 13 pathologies or the request/response contract): gateway's `GET /`
  serves the frontend rather than §4's generic info JSON (the other five
  HTTP services still return that JSON as specified); `insights` splits a
  cheap PK-only `quick_summary` (embedded in every transaction response)
  from the real cached/N+1-affected 120-day analytics endpoint, so the
  missing-index pathology doesn't tax every request; `retry_storm` fires
  all 4 attempts unconditionally rather than stopping at first success
  (spec's "no circuit breaker" taken literally — also the only way to reach
  the required lookup-multiplication ratio given `sanctions_errors`' fixed
  3% rate); `poison_data` resolves via a Redis hash populated at seed time
  instead of a live table scan, to avoid the deliberately missing index;
  uvicorn's access log is disabled fleet-wide (redundant with the spec's
  own logging design, and was double-counting two log-line-based pathology
  tests); Postgres `max_connections` raised from 100 to 300 (five
  services' pool sizes can sum past the default under load).
 **Time-box:** 2–3 evenings. Two agent sessions plus one evening of your own
 reading.
 **Deliverable:** the working Ledgerline system with no observability, plus a
 black-box integration suite that pins down both its working state and its
 deliberate brokenness.
 ### Why TDD here, when the app itself doesn't matter
 The microservices are throwaway. The *properties* aren't. Everything from Phase 2
 onward assumes the system still behaves a specific way: seven real services in
 the path, async work genuinely off the request path, five hot accounts owning a
 third of the rows, and thirteen pathologies that actually degrade something. You
 will be editing all seven services for weeks. Without a suite that fails when one
 of those properties quietly breaks, you'll spend an evening chasing a trace
 anomaly that turns out to be a regression you introduced yesterday.
 Writing the tests first also forces the contract to be explicit before anything
 is built — which is what makes a second agent able to implement against it
 without guessing.
 ### 0a — the test suite
 Agent session one, from `phase-0a-test-suite-spec.md`. That agent writes only
 `tests/` and must not create so much as a stub under `services/`. Its output
 should be **red**, with a `tests/RED-BASELINE.md` recording why each test fails
 right now — a test that passes against an empty repo is a broken test.
 The suite asserts goals, not mechanics. Two representative examples worth
 understanding before you read the code:
 - *"Stop the `sanctions` container and transactions must fail."* This is the test
  that catches an implementation quietly collapsing seven services into one
  process. Nothing else would.
 - *"With `n_plus_one` on, one insights request issues ≥ 50 SQL statements."*
  Measured via `pg_stat_statements`, not wall clock — because a query count
  can't be faked with a `sleep()`, and a latency threshold can.
 That second point is the design principle throughout: **prefer assertions that
 are hard to satisfy without the real mechanism.** Query counts, `EXPLAIN` plan
 node types, Redis consumer-group lag, container RSS, and genuine unhandled
 exception paths all have that property.
 ### 0b — the implementation
 Agent session two, from `phase-0-base-project-spec.md`, with `tests/` handed over
 as a **read-only contract**. That agent may not edit, skip, `xfail` or loosen a
 single test. If something looks impossible it stops and reports rather than
 adjusting the target.
 Done when `make test` is green three consecutive runs and
 `git diff --stat tests/` is empty.
 ### Then, yourself
 1. **Read every service's `app.py` end to end.** You need to know this codebase
   cold, because from Phase 2 onward you're modifying all seven services.
 2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
   then try to figure out what's wrong using only `docker compose logs`.** Give it
   twenty real minutes. You will fail, and the specific way you fail — no timing,
   no causality, no cross-service correlation, a `X-Request-Id` that dies at the
   Redis Stream — is the motivation for everything that follows. Write down what
   you wished you had. That list is your acceptance criteria for Phase 8.
 **Done when:** `make test` is green three runs in a row, all nine acceptance
 criteria in the base spec pass, and you have that written list of frustrations.
@@ -0,0 +1,89 @@
 # Phase 1 — The model, and one signal end to end
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
 **Status:** Not started
 **Message:** Blocked on nothing — Phase 0 is done and the platform is
 described above. Waiting to begin.
 **Time-box:** one evening.
 **Why:** you need the vocabulary, but more importantly you need to see one trace
 appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
 than abstract.
 ### Concepts
 - **Monitoring vs observability.** Monitoring answers questions you knew to ask
  (known unknowns): dashboards and alerts built from predefined aggregates.
  Observability is a property of the system: can you ask a *new* question about
  production, that you never anticipated, without shipping new code? Charity Majors'
  framing — monitoring is known-unknowns, observability is unknown-unknowns. The
  practical implication for you: emit **wide, high-dimensional events** (rich span
  attributes), not just pre-aggregated counters.
 - **The OTel architecture split: API vs SDK vs Collector.**
  - The **API** is what your code and third-party instrumentation libraries call.
    If no SDK is configured, API calls are no-ops — that's deliberate, it means a
    library can be instrumented without forcing a dependency on any implementation.
  - The **SDK** is the engine: sampling, batching, context propagation, export.
    Application owners configure the SDK; library authors only touch the API.
  - The **Collector** is a standalone process that receives, processes and exports
    telemetry. Optional, but you should use one.
 - **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
 - **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
  `service.version`, `deployment.environment`) and is attached to every signal from
  that process. Attributes describe the individual span/metric/log.
 - **Semantic conventions**: standardised attribute names (`http.request.method`,
  `db.system`, `server.address`). Use them for anything standard; namespace your own
  (`ledgerline.account.id`). This is what makes cross-service, cross-language
  correlation work — and it's what Grafana's prebuilt dashboards key off.
 - **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
  aware of, not learning yet.)
 - **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
  app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
  seven services. The Collector also does the things you don't want in your app:
  retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
  destinations.
 ### Resources
 - freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
  M.E.L.T. and context-propagation sections; skip the Node code.
 - OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
 - Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
  repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
  it — just see what "done" looks like.
 ### Build
 1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
   `ansible-galaxy collection install -r requirements.yml`, change
   `grafana_admin_password` (vault it), then
   `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
   README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
   Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
   Then open the Alloy UI and **read the component graph** — it shows you the exact
   pipeline, including the fact that the OTLP receiver has only a traces output.
   Seeing that in the UI before you get bitten by it is worth ten minutes.
   *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
   a single container with everything pre-wired. Useful for a first trace, but
   don't build on it — its pipeline differs from your real platform in exactly the
   ways that matter.)*
 2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
   auto-instrumentation:
   ```
   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
   ```
   and change its command to
   `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
   with `OTEL_SERVICE_NAME=sanctions`,
   `OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
   `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
   Tempo directly.
 3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
 **Done when:** you can see a single-service trace in Tempo, and you can explain out
 loud what the Resource attributes on it mean and where they came from.
@@ -0,0 +1,40 @@
 # Phase 10 — Package the delivery
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)*
 **Status:** Not started
 **Message:** Depends on every prior phase being done — this is the wrap-up,
 not independent work.
 **Time-box:** one evening.
 Babak gave you a task; deliver it like an engineer, not like a student.
 1. **`README.md`** — architecture diagram, one-command start, what each container is
   for, and a "here's the demo" script.
 2. **The five-minute demo path**, rehearsed:
   - Delete every dashboard in the Grafana UI, then `ansible-playbook
     playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
     from code, one command.
   - Submit one transaction from the frontend, then follow it through Grafana: RED
     dashboard → latency spike → exemplar → trace → span → logs for that span → back
     to the trace. One request, all three signals, no searching.
   - `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
     and the service map turn red at a specific node.
   - Show `docs/INVESTIGATION.md` and how you found the pathologies.
 3. **The written answers** Babak asked for, as documents in the repo, not as a
   conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
   `docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
   RTO/RPO/SLA/SLO/error budget in your own words.
 4. **The test suite as part of the story.** Show `make test` green, then show the
   three generations of the call-counting assertion side by side —
   `docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
   each signal bought you. Most people demo dashboards; almost nobody can show
   their observability work paying off inside their own test suite.
 5. **A "what I'd do differently in production" section.** Tail sampling instead of
   100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
   log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
   you *didn't* do, and why, is the strongest signal of seniority in the whole
   delivery.
@@ -0,0 +1,116 @@
 # Phase 2 — Distributed tracing
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
 **Status:** Not started
 **Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
 **Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
 **Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
 (Resource, providers, exporters, Collector wiring) is shared across all three
 signals — do it once here and logs and metrics become incremental. Second, tracing
 is what Babak actually asked for ("trace a single request, see where it stuck"),
 and once trace IDs exist, log correlation in Phase 3 works on the first try instead
 of needing rework.
 ### Concepts
 - **Trace / span / span context.** A trace is one request's journey; a span is one
  timed operation within it. Each span carries a trace ID, its own span ID, and its
  parent's span ID. That parent pointer is the entire data structure.
 - **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
  outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
  these right is what makes the service graph render correctly.
 - **Context propagation.**
  - *In-process*: OTel uses Python `contextvars`. This works across `await` — but
    **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
    carry the context explicitly.
  - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
    `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
  - The classic bug: any boundary that doesn't carry those headers **starts a brand
    new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
    `TraceContextTextMapPropagator().inject(...)` into the message payload on the
    producer and `.extract(...)` on the consumer, then starting the consumer span
    with that context as parent. This is the single most valuable exercise in the
    phase — most people never do it manually and never really understand it.
  - Related: `links` (for a consumer span that batches messages from many traces)
    and **Baggage** (arbitrary key/values propagated alongside trace context —
    useful, and a foot-gun, because it's sent on every hop and is visible to
    downstream systems; never put secrets in it).
 - **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
  client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
  gets you the business semantics: `score_transaction`, `check_sanctions_list`,
  `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
  that show *which service* was slow but not *why*.
 - **Span attributes vs metric labels.** Spans are the right home for
  high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
  with volume, not with distinct values. This is why "put it on the span, not the
  metric" is the standing rule.
 - **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
  That's how the exception ends up attached to the trace and how the `spanmetrics`
  connector in Phase 4 will count your error rate.
 - **Sampling — the concept now, the config later.**
  - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
    errors at the same rate as successes.
  - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
    traces, but must run in the Collector and buffer spans in memory.
  - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
    now; you'll configure tail sampling in the Collector in Phase 5.
  - Whatever you choose, the sampling decision must **propagate** — the sampled flag
    rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
    unilaterally drop half of a trace its caller decided to keep.
 ### Resources
 - Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
  min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
 - Your chosen Udemy backbone: the instrumentation + context-propagation sections.
 - SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
  practical FastAPI-specific setup, including batch/queue env vars.
 - OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
  reading.
 - **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
  side by side. Look specifically at how they mix auto-instrumentation with
  hand-rolled business spans, and how the Kafka producer/consumer propagate context.
  This is the "professional implementation" you asked to study.
 ### Build
 1. **Shared bootstrap.** Create `common/telemetry.py` with a single
   `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
   `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
   service calls it once at startup. Keep it under 60 lines.
 2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
   `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
   you now get a connected trace across `gateway → ledger → fraud → sanctions`.
 3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
   `sanctions.lookup_watchlist`, `insights.aggregate_120d`,
   `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
   namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
   `ledgerline.tx.amount_minor`).
 4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
   inject trace context into the message fields. Consumers (`notifier`, `audit`):
   extract, and start a `CONSUMER` span with the extracted context as parent.
 5. **Record exceptions** everywhere you currently `logger.exception(...)`.
 6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
   more.
 7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
   the "before" state so you could measure against it, and from here it's
   actively wrong. Everything else in `tests/` must still pass: instrumenting
   the system is not allowed to change its behaviour, and if `make test` goes red
   after adding spans, you've broken something real. That is exactly the safety
   net you built it for.
 ### Done when
 - One request produces **one connected trace** covering all seven services,
  including both async consumers.
 - With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
  ate the time.
 - You can explain why `blocking_event_loop` looks different in a trace than
  `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
  before spans start, plus every concurrent request in that process degrading
  together.)
@@ -0,0 +1,99 @@
 # Phase 3 — Structured logs, correlated to traces
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
 **Status:** Not started
 **Message:** Depends on Phase 2 (trace IDs must exist before log correlation
 means anything).
 **Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
 ### Concepts
 - **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
  events: a stable event name plus typed fields (`event="transaction_settled"`,
  `account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
  Consistent field names across all seven services — `account_id` everywhere, never
  `accountId` in one place and `acct` in another. This is what makes LogQL queries
  possible at all.
 - **Trace–log correlation.** Every log line emitted inside an active span must carry
  `trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
  "one pane of glass": click a span → see its logs; click a log line → open its
  trace. Both directions are configured, and you'll set both up.
 - **Two ways to ship logs, and which one your platform actually does:**
  - **stdout → collector scraping → Loki.** The container-native way. Your app just
    prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
    standard in Kubernetes. **This is the only path wired in your Alloy config**
    (`loki.source.docker` reading the Docker socket).
  - **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
    with the same Resource attributes attached automatically. Cleaner correlation,
    but the app owns delivery. **On your platform this currently goes nowhere** —
    Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
    dropped in silence.
  - Do both anyway: use stdout for most services, and *extend Alloy* to wire the
    OTLP logs path for one or two. Then argue the trade-off from experience. That
    comparison — including the fact that a misconfigured pipeline fails silently
    rather than loudly — is precisely what Babak will probe.
 - **The field name is not a style choice here.** Your Loki datasource is provisioned
  with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
  A log line must carry lowercase `trace_id`. OpenTelemetry's logging
  auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
  logs and traces both arriving, no errors anywhere, and no clickable link. Hence
  structlog with an explicit processor rather than the auto-injection shortcut.
 - **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
  your entire fleet, and Loki bills/indexes on volume.
 - **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
  `json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
  redaction into the processor chain, not into each call site.
 - **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
  not log content. Labels should be few and low-cardinality (`service`, `env`,
  `level`). Everything else lives in the log line body and is filtered at query time.
  Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
  in the JSON body and use a **derived field** to make it clickable.
 ### Resources
 - *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
  Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
  setup, including the hex formatting for trace/span IDs.
 - OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
  `OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
  fields, if you'd rather not hand-roll a processor.
 - Udemy backbone: the Loki + LogQL sections.
 - Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
 ### Build
 1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
   processor that pulls the current span context and injects `trace_id` / `span_id`.
 2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
   handler: request received (or rely on the span), business decision, error.
 3. Add a redaction processor with a denylist of field names.
 4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
   with a `container` label (Alloy's relabel rule sets it from the Docker container
   name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
 5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
   send some records, and watch nothing appear in Loki. Then check the Alloy UI:
   the receiver shows traffic, and the pipeline shows no logs output. Now extend
   `config.alloy.j2` — add a `logs` output on the receiver plus an
   `otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
   them arrive. Write down what you'd have done to debug this if you *hadn't* known.
   That is the most transferable half-hour in the whole project.
 6. Confirm the provisioned correlation works in both directions: Loki's
   `derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
   (`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
   are verifying, not writing. If a link doesn't appear, the regex didn't match
   your field name — check that before anything else.
 6. **Migrate the test suite's log-counting helpers.** Three pathology tests
   (`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
   in `docker compose logs` — the only cross-service call counter that existed at
   Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
   with LogQL over structured fields instead. This is a small change and a
   pointed one: notice how much shorter and more precise the assertion gets, and
   that it now works without shelling out to Docker at all. That delta is the
   value of structured logging, measured on your own code.
 **Done when:** from a slow trace you can click into that request's logs, and from
 any log line you can click into its trace — in both directions, in two clicks.
@@ -0,0 +1,126 @@
 # Phase 4 — Metrics
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
 **Status:** Not started
 **Message:** Not blocked technically (metrics don't strictly need tracing
 first), but sequenced after Phase 3 per the roadmap's ordering.
 **Time-box:** 2 evenings.
 ### Concepts
 - **Instrument types.**
  - **Counter** — monotonic, only increases. Requests, errors, messages consumed.
  - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
    connection-pool checkouts.
  - **Histogram** — a distribution. Request latency, payload size. This is what
    gives you percentiles.
  - Synchronous instruments are recorded inline; **observable/asynchronous** ones are
    read via a callback on collection (right for things you *poll*, like pool size).
  - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
    **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
    nonsense. Use histograms.
 - **Cardinality — the one thing that will bite you.** Every unique combination of
  metric name + label values is a separate time series in memory. Add `account_id`
  as a label on a 800-account demo and you've made 800 series; do it in production
  and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
  ~10 000 distinct values in production, it does not belong on a metric.** Put it on
  a span. Histograms multiply this (one series per bucket per label combination), so
  don't over-bucket either.
  - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
    the concrete path), `http.status_code` class, `kind`, `currency`.
  - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
    raw URL, timestamp, error message.
 - **RED and USE.**
  - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
    per-service dashboard.
  - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
    connection pools, queues, disk. Saturation is the one everyone forgets and the
    one that predicts collapse.
  - You want both: RED tells you the service is slow, USE tells you which resource
    is why.
 - **Span metrics — generate metrics from traces.** Two architectures exist for this,
  and knowing both is the point:
  - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
    `servicegraph` connectors read spans in flight and emit RED metrics and edge
    metrics. This is what most blog posts (including the OneUptime article) show.
  - *In the trace store:* Tempo's **metrics-generator** does the same job after
    ingestion, remote-writing the result to Prometheus. **This is what your platform
    does** — `processors: [service-graphs, span-metrics, local-blocks]` with
    `send_exemplars: true`, remote-writing to a Prometheus started with
    `--web.enable-remote-write-receiver`.
  - So RED metrics, the service map and exemplars **already exist** before you write
    a single metric line. Do not add the collector connectors as well — you'd
    duplicate every series under a second set of names. **Go find the real metric
    names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
    they will not match the article's.
  - The trade-off worth being able to argue: collector-side generation is cheaper
    and works before storage, but sees only what passes through that collector;
    Tempo-side generation sees everything that landed, survives collector restarts,
    and can be enabled per-tenant — at the cost of load on the trace store.
 - **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
  click a spike on a latency graph and land in the exact trace. On your platform
  this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
  and the Prometheus datasource declares `exemplarTraceIdDestinations` with
  `name: trace_id`. If you emit your own histograms with exemplars, the label must
  be `trace_id` to match.
 - **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
  can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
  for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
  the pull model is why Prometheus has service discovery and why `up` exists.
 ### Resources
 - Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
  Read the first half. Best cardinality explanation available.
 - Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
  not a fact.
 - **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
  `servicegraph` connector config and its RED PromQL queries almost verbatim.
 - OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
 ### Build
 1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
 2. Custom application metrics, deliberately few:
   - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
   - `ledgerline.transaction.duration` (Histogram; labels: `kind`)
   - `ledgerline.fraud.score` (Histogram; no labels)
   - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
   - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
 3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
   metrics, so pick a path and implement it:
   - *Platform grain:* expose `/metrics` on each service and add them to
     `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
     playbook. Prometheus scrapes your services directly. Note the irony and the
     lesson — you're back to a pull model and a second client library, which is
     exactly the fragmentation OTel exists to fix.
   - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
     Alloy so app metrics flow through the collector as the architecture drawing
     intends. This also closes a real diagram-vs-code gap in the repo.
   - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
     `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
     all three signals.
   Do the first to get moving; do one of the others afterwards and write up which
   you'd choose for a real platform team and why. That write-up is a Phase 9
   artefact.
 4. Deliberately break it: add `account_id` as a label to
   `ledgerline.transactions.total`, run a 200-request burst, and watch the series
   count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
   you'll never forget.
 5. Compare your hand-written duration histogram against the `spanmetrics`-generated
   one. Understand why they differ slightly and when you'd want each.
 6. **Second migration of the call-counting assertions.** The three tests you moved
   to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
   TraceQL query for `sanctions` client spans within a single trace answers
   "how many lookups did this one transaction make?" directly, which is the
   question the test was always really asking. Do it and compare all three
   generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
   clearest possible demonstration of what each signal is actually for, on a
   problem you already understand.
 **Done when:** you have a RED view per service, a working service map, and you can
 click a latency spike through to a trace via an exemplar.
@@ -0,0 +1,76 @@
 # Phase 5 — Dashboards and alerts as code
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
 **Status:** Not started
 **Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
 dashboarding.
 **Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
 recreate in seconds.*
 ### Concepts
 - Three levels of "as code", and you should know when each applies:
  1. **File provisioning** — YAML in `provisioning/datasources/` and
     `provisioning/dashboards/`, plus dashboard JSON files mounted into the
     container. Grafana loads them at startup. Simplest, no state, no API tokens,
     perfect for a Compose demo. **This alone satisfies the requirement.**
  2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
     PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
     function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
     diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
     of Grafonnet (which is legacy and unofficial).
  3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
     multi-environment, or GitOps setups. Overkill here; the right answer *later*,
     when this lands on your Kubernetes cluster.
 - **Dashboards should answer questions, not display data.** A good service dashboard
  answers, in order: is it up, is it serving, is it erroring, is it slow, and where
  is the pressure. Panels that nobody looks at during an incident are cost, not value.
 - **Alerts as code too.** Grafana alert rules, contact points and notification
  policies all provision from YAML. And keep the *alert definition* next to the
  dashboard that explains it.
 ### Resources
 - Grafana docs → *Observability as code* (the index page — it lays out Foundation
  SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
 - Grafana Foundation SDK repo → the Python examples, especially the `red-method`
  example.
 - Udemy backbone: dashboard design sections (panel types, transformations,
  variables, dashboard best practice).
 ### Build
 Your platform provisions **datasources only** — there is no dashboards provider at
 all. So this phase is entirely yours, and the right home for it is inside the
 Ansible role, using the same mechanism as everything else.
 1. **Extend the role.** Add
   `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
   (a file-based dashboard provider pointing at a mounted directory), a
   `config/grafana/dashboards/` directory for the JSON, and the tasks to template
   and copy them. The Compose template already mounts
   `./config/grafana/provisioning` read-only, so the provider is picked up on
   restart.
 2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
   Foundation SDK (Python)**: one function producing a RED dashboard for a given
   service name, looped over the seven services; a second producing a system
   overview (service graph, stream lag, pool saturation, error-budget burn). Commit
   the generator *and* its output; the Ansible role ships the output.
 3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
   Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
   That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
   than a local Makefile target, because it's the same command that builds the
   platform.
 4. While you're in there, review the provisioned datasources and add what's missing:
   `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
   worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
   understand why `filterByTraceID: true` alone is sufficient here, and when it
   wouldn't be.
 5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
 **Done when:** re-running the playbook restores every dashboard from scratch, and no
 dashboard has ever been saved through the Grafana UI.
@@ -0,0 +1,94 @@
 # Phase 6 — Load testing with k6, and percentiles
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
 **Status:** Not started
 **Message:** Depends on Phase 4 (metrics to plot k6's output against) and
 benefits from Phase 5's dashboards existing.
 **Time-box:** 1–2 evenings.
 ### Concepts — P95 and P99 properly
 You asked about these specifically, and there's more to them than "the 95th
 percentile."
 - **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
  is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
  is.
 - **You cannot average percentiles.** Not across instances, not across time buckets.
  `avg(p99)` is wrong and produces a number that looks plausible. The correct move is
  to aggregate the *histogram buckets* and compute the quantile from the sum:
  `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
 - **Prometheus classic histograms give you an interpolated estimate**, bounded by
  your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
  bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
  bracket your actual SLO threshold. (Native histograms largely solve this and are
  worth knowing about.)
 - **Which population?** An overall p99 across all endpoints is dominated by whichever
  endpoint has the most traffic. Slice by route, or you'll miss that
  `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
 - **Tail latency amplification** — the most important idea here for a microservice
  system. If one user request fans out to 10 backend calls, each with an independent
  p99 of 1 s, then the chance the *user request* touches at least one slow call is
  `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
  p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
  fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
  measurable — go measure it.
 - **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
  most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
  that.
 - **Coordinated omission** — a load-generator trap. If your generator waits for a
  response before sending the next request, then when the system stalls, the
  generator stalls too and simply *doesn't record* the requests that would have been
  slow. Your measured latency is systematically optimistic. Fix: use an **open
  model** — a fixed *arrival rate* independent of response times. In k6 that means
  the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
  `constant-vus`. Knowing this is a genuine senior-level differentiator.
 ### Concepts — k6
 - **Test types:** smoke (does it work at all), average-load (normal traffic),
  stress (past normal, find the knee), spike (sudden surge, test recovery),
  soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
  Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
 - **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
  (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
 - **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
  fail the run; **thresholds are SLOs expressed in code and they set the process exit
  code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
 - **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
  view and the application's view sit on the same Grafana time axis. That side-by-side
  is the single most convincing screenshot you can show Babak.
 - k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
  writing 60 lines of it.
 ### Resources
 - Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
  *Thresholds*. Free, genuinely well written; ~2 h to work through.
 - Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
  hands-on).
 - Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
 - Grafana docs → *k6 output to Prometheus remote write*.
 ### Build
 1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
   sharing a `common.js` for setup and payload generation against random accounts.
 2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
 3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
   `http_req_failed rate<0.01`.
 4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
 5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
   sets. This before/after pair is your baseline for Phase 8.
 6. Optional but impressive: have k6 send W3C trace context headers so load-test
   requests appear as traces, letting you click from a k6 spike into a specific
   trace. Check the current k6 docs for the tracing module's status — it has moved
   between experimental and stable.
 **Done when:** `make load-stress` produces a Grafana view where you can see the
 arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
 service that broke first.
@@ -0,0 +1,91 @@
 # Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
 **Status:** Not started
 **Message:** Depends on Phase 6's load-test baseline (the latency SLO should
 come from measured numbers, not be picked from thin air).
 **Time-box:** 1–2 evenings. Mostly thinking, a little config.
 ### Concepts
 - **SLI** — a *measurement* of something users experience, expressed as a ratio of
  good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
  `POST /api/transactions` requests that returned 2xx within 500 ms.*
 - **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
  requests succeed within 500 ms, measured over a rolling 28 days."
 - **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
  Always **looser** than your SLO, so you have room to miss internally without
  breaching commercially. If your SLA and SLO are the same number, you've made a
  mistake.
 - **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
  budget; over 30 days that's about **43 minutes**. The budget is the thing that
  turns reliability from an argument into arithmetic: if there's budget left, ship
  features and take risks; if it's spent, the policy says you stop feature work and
  spend on reliability. That policy — agreed in advance by dev, ops and product — is
  the whole point.
 - **Burn rate.** How fast you're consuming budget relative to the window.
  Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
  multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
  is what stops you from either paging on every blip or noticing the slow bleed a
  month late.
 - **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
  and it's worth being precise because interviewers probe exactly here.
  - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
    from failure to restored service. It drives your architecture: replication,
    warm standby, failover automation, runbook quality.
  - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
    in time: how far back can you afford to lose. It drives your **backup and
    replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
    every 15 minutes.
  - The relationship worth stating out loud: **SLO governs steady-state quality; RTO
    and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
    "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
    5 minutes of transactions." Both feed the SLA.
  - Both are **business decisions, not technical ones** — you derive them by asking
    what an hour of downtime or an hour of lost data actually costs, then buying the
    cheapest architecture that meets it. Tightening either is expensive and
    non-linear.
 ### Resources
 - Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
  immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
 - Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
  all free PDFs. Run the exercises solo with Ledgerline as the subject.
 - Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
  free, self-paced, deeper.
 - For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
  overview; pair it with your Postgres backup knowledge.
 ### Build
 1. Write `docs/SLO.md`. For Ledgerline define, with justification:
   - Availability SLI/SLO on `POST /api/transactions`
   - Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
     from thin air)
   - A freshness/lag SLI on the notifier consumer group
   - A stated SLA that is deliberately looser than each SLO
   - An RTO and RPO for the Postgres instance, with the backup cadence they imply
   - An error budget policy: what happens at 50%, 75% and 100% consumed
 2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
   Your platform ships **Alertmanager** plus a Prometheus rules directory and the
   sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
   *Prometheus rule files → Alertmanager*, templated by the Ansible role, with
   `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
   be able to argue the alternative: Grafana unified alerting can evaluate across
   datasources (including Loki queries) and lives with your dashboards, whereas
   Alertmanager is metrics-only but is the ecosystem standard and gives you
   grouping, deduplication, inhibition and silences for free. Note which you chose
   and why in `docs/SLO.md`.
 3. Add an error-budget panel to the overview dashboard: budget remaining, current
   burn rate, projected exhaustion.
 4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
   watch the fast-burn alert fire.
 **Done when:** you can explain, in one paragraph each and without notes, the
 difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
 decision.
@@ -0,0 +1,90 @@
 # Phase 8 — Bottleneck hunting (the capstone)
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
 **Status:** Not started
 **Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
 dashboards, load-test baselines, and burn-rate alerts are the toolkit this
 phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
 **Time-box:** 2 evenings. This is where the whole thing pays off.
 ### Concepts — how to actually answer "how do I find bottlenecks?"
 Babak asked how you'd *think* about this, not for a list of tools. Here's the
 method, roughly in order. The discipline is to go **top-down from the user, not
 bottom-up from the infrastructure** — bottom-up is how people spend three hours
 tuning a database that wasn't the problem.
 1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
   window, compared to what baseline? "It's slow" is not a starting point. "p95 of
   `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
 2. **Localise to a service** using RED + the service graph. Which node's duration
   grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
   the graph from the leaves inward.
 3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
   TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
   several slow traces plus one fast trace of the same endpoint as a control.
 4. **Read the trace for five specific shapes:**
   - **A long span** — the work itself is slow. Look at its attributes: which query,
     which host, which parameters.
   - **A gap** — wall-clock time not covered by any span. This is the interesting
     one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
     blocked event loop, or a missing instrumentation library. Ledgerline's
     `pool_starvation` and `blocking_event_loop` both show up as gaps.
   - **Serialisation** — spans laid end to end that could have been concurrent.
     Ledgerline's `no_batching` looks exactly like this.
   - **Repetition** — the same span name 150 times in a row. That's your N+1.
   - **Retries** — the same client span repeated 3–4 times with errors. That's
     `retry_storm`, and note that it makes an *overload* worse, never better.
 5. **Separate latency from saturation.** Switch from RED to USE on the suspected
   resource: connection pool in-use vs pool size, stream consumer lag, CPU,
   memory. A resource at 100% utilisation with a growing queue is saturated;
   latency there is a *symptom*, not the cause.
 6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
   equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
   concurrent slots. If your pool is 5, you queue — and queueing latency grows
   non-linearly as you approach saturation. This one formula explains most
   connection-pool and worker-count incidents you'll ever see.
 7. **Check for tail amplification** (Phase 6) before blaming any single service.
 8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
   chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
   over the *same duration*. One variable at a time. Never declare a fix from a
   dashboard that "looks better."
 9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
   becomes the bottleneck. Re-measure after every change; stop when you're inside
   the SLO, not when you run out of ideas.
 10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
    or lock waits inside Postgres, or which line of Python burned the cycles. That's
    where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
    profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
 ### Resources
 - Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
  for saturation.
 - Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
 - *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
  thing ever written on why microservice latency behaves the way it does.
 - Grafana docs → Tempo → **TraceQL**. You'll live in this.
 - Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
  these; connect them to the trace's DB spans.
 ### Build
 1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
   `docs/PATHOLOGIES.md`.**
 2. Run `make load-average` for 10 minutes.
 3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
   like a real incident doc: what you observed, what you hypothesised, what you
   checked, what it turned out to be. Include screenshots.
 4. When you think you've found them all, run `make chaos-clean`, enable them one at
   a time, and confirm each signature.
 5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
 6. Repeat once with a new random set. The second run is where the method actually
   becomes yours.
 **Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
 and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
@@ -0,0 +1,161 @@
 # Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
 **Status:** Not started
 **Message:** Not technically blocked by earlier phases (it stands on its own
 as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
 re-diagnose in Kibana already exists.
 **Time-box:** 1–2 evenings.
 Babak's remaining questions are all *judgment* questions. He isn't testing whether
 you can recite what Filebeat does; he's testing whether you have a framework for
 choosing. Here's the framework, then the answers as worked examples.
 ### The framework — six axes to reason on
 Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
 1. **Where does the data originate?** Inside your application (you can instrument it)
   or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
   touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
 2. **What shape is it already in?** Structured JSON at the source, or unstructured
   text needing grok/regex parsing? Parsing cost has to land *somewhere*.
 3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
 4. **What's the destination, and how many?** One backend forever, or several, or one
   you expect to change?
 5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
   on every host) distributes cost and adds resilience but multiplies config surface.
   Parsing centrally concentrates cost and creates a single point of failure — but
   one place to change rules.
 6. **What already exists, and what does it cost?** Existing Elastic investment,
   existing licences, team skills, and licensing constraints (Elastic's Filebeat and
   Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
   Bit and Vector are permissively licensed). Migration is never free.
 ### The answers
 **When to use Filebeat.** When you need a lightweight shipper that tails log *files*
 from a host or container and forwards them with minimal transformation — and your
 destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
 delivery with a registry of file offsets, backpressure handling, and prebuilt modules
 for common services (nginx, MySQL, system logs) that come with parsing and dashboards
 included. Reach for it when the source is a file you don't control the format of, the
 app can't be modified, and Elasticsearch is where it's going.
 **When to use Logstash.** When you need real *transformation*: parsing unstructured
 text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
 splitting or merging events, conditional routing to multiple destinations, or
 buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
 for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
 per host.
 **When to use Filebeat and Logstash together, and why.** This is the classic Elastic
 architecture and the reason is a **division of labour between edge and centre**:
 - Filebeat runs on *every* host: cheap, tails files, handles offsets and
  backpressure, ships onward. You want the thing running in a thousand places to be
  small and boring.
 - Logstash runs on a *few* central nodes: does the expensive grok parsing,
  enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
  you want them in a handful of places you can redeploy, not on a thousand hosts.
 - It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
  persistent queue absorbs it and Filebeat backs off gracefully, rather than every
  host queuing independently or dropping.
 So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
 Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
 need central buffering. Skip Logstash when your logs are already structured JSON and
 Elasticsearch ingest pipelines can do the little parsing that remains.
 **When to use OpenTelemetry.**
 - You own the application code and want telemetry generated *from inside it* — spans
  with business context, custom metrics, logs already carrying trace IDs.
 - You need **more than one signal**, correlated. This is the strongest argument: one
  SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
  threaded through all three. Neither Filebeat nor Logstash does traces.
 - You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
  data model across all of them.
 - You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
  Jaeger by editing Collector YAML.
 - You need distributed tracing at all. There is no serious vendor-neutral alternative.
 - Your telemetry needs processing in flight — redaction, tail sampling, attribute
  rewriting — without touching the app.
 **When *not* to use OpenTelemetry.** This is the question that separates people who
 have used it from people who have read about it.
 - **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
  an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
  file-based collector.
 - **You only need one signal from one source with no correlation.** A single nginx
  access log going to a single log store: OTel is more machinery than the job needs.
  Fluent Bit or Filebeat is simpler and finishes the job today.
 - **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
  Elastic estate with hundreds of Beats modules, dashboards and detection rules is
  years of investment. "We should use OTel because it's the standard" is not a
  business case. Note the direction of travel though: the Collector can already
  receive from and export to most of these, so a *gradual* migration is usually the
  right shape — often by putting a Collector or Vector in front of the existing
  agents rather than ripping them out.
 - **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
  where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
 - **The signal is genuinely business analytics, not telemetry.** Clickstream data for
  a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
  retention, different query patterns, different cost model.
 - **Maturity gaps for a specific need in a specific language.** OTel logs support has
  matured at different rates per language, and profiling is still alpha. Check the
  status page for your language before betting on a signal.
 - **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
  on it. A two-person team that needs error tracking by Friday should install Sentry.
 The senior framing to give Babak: *these tools aren't competitors so much as
 different layers.* OpenTelemetry is a **generation and transport standard** for
 telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
 don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
 Elasticsearch are **stores**. Most real architectures use one from each layer, and
 the interesting question is always which layer a given problem lives in.
 ### Resources
 - Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
 - SigNoz — *OpenTelemetry vs ELK Stack* (free article).
 - OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
  read these to see how the Collector does the Filebeat/Logstash job itself.
 - OpenTelemetry docs → *Instrumentation status by language* — for the maturity
  caveats.
 ### Build — and this is no longer hypothetical
 Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
 deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
 Filebeat reading Docker container logs and Logstash parsing JSON into daily
 `logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
 questions — the codebase he handed you contains both worlds.
 1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
   flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
   Alloy → Loki. Compare them on the same data.
 2. Test the direct ingest path the README documents — pipe a JSON line into
   Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
   parsing happens before you change it.
 3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
   that parses them. Then do the same job with Alloy's `loki.process` stages. Time
   both. Note which one you'd rather debug at 2 a.m.
 4. Compare the *investigation experience* directly: take one incident you already
   diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
   you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
   wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
   for some questions, and Loki's cost model genuinely beats Elasticsearch's for
   others.
 5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
   licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
   permissively licensed. For some employers that alone decides it.
 6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
   valuably — what you actually found annoying about each. Having run both stacks
   against the same application is the difference between reciting a comparison and
   arguing one.
@@ -0,0 +1,71 @@
 # Roadmap summary — all phases at a glance
 *Part of `docs/roadmap/` — the one-page overview. Read [`intro.md`](intro.md)
 first for platform/background context; read the individual `phase-N.md` file
 for the phase you're actually working on. This file exists so an agent (or
 you) can check overall status and sequencing without loading every phase's
 full detail.*
 Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase
 file follows the same shape: **Status**, **Message** (what's done, notes,
 warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
 ## Status and focus
 | Phase | File | Status | Focus | Time-box | Main output |
 |---|---|---|---|---|---|
 | 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
 | 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
 | 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
 | 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
 | 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
 | 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
 | 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
 | 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
 | 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
 | 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
 | 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
 | 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
 ## What each phase is, in one line
 - **0 — Build the base project, test-first.** Write the black-box test suite,
  then implement the seven-service system against it as a frozen contract.
  **Done** — see `phase-0.md` for the full implementation report.
 - **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
  split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
  Ansible and get one auto-instrumented trace into Tempo.
 - **2 — Distributed tracing.** The heart of the task: auto- and
  manual-instrument all seven services, and hand-propagate trace context
  across the Redis Stream hop, which carries none by default.
 - **3 — Structured logs, correlated to traces.** Replace prose logging with
  structured events carrying a lowercase `trace_id`, so Loki and Tempo link
  in both directions.
 - **4 — Metrics.** Instrument RED metrics per service, understand cardinality
  limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
  silently. Tempo's metrics-generator already produces span metrics and the
  service graph — don't duplicate it in the collector.
 - **5 — Dashboards and alerts as code.** Generate dashboard JSON with the
  Grafana Foundation SDK and ship it through the Ansible role, so "destroy
  and recreate in seconds" is a real, repeatable command.
 - **6 — Load testing with k6, and percentiles.** Learn why percentiles can't
  be averaged, what coordinated omission is, and drive Ledgerline with
  arrival-rate executors to get a clean before/after load profile.
 - **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
  SLOs from the Phase 6 baseline (not from thin air), write an error-budget
  policy, and wire multi-burn-rate alerts.
 - **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
  of chaos flags, find the pathologies from telemetry alone using a top-down,
  hypothesis-driven method, and write it up as an incident doc.
 - **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
  OpenTelemetry on six concrete axes, using the platform's real second stack
  (Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
 - **10 — Package the delivery.** README, a rehearsed five-minute demo, the
  written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`,
  `CONCEPTS.md`), and an honest "what I'd do differently in production"
  section.
 ## If you have to cut scope
 Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not
 cut Phase 8** — it's the only phase that proves you learned anything.