Add deployment phase and Traefik compose override

Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-09 15:38:29 +03:30
parent b2112e55e1
commit 5e514ecf25
17 changed files with 1080 additions and 937 deletions
@@ -16,9 +16,9 @@ Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is
 assessed by a mentor, Babak.
 The full plan lives in `docs/roadmap/`, split one file per phase so an agent
-working on a given phase doesn't have to load the other ten: `intro.md`
+working on a given phase doesn't have to load the other eleven: `intro.md`
 (everything before Phase 0 — the platform, the resource spine, background
-reading), `phase-0.md` through `phase-10.md`, and `summary.md` (a one-page
+reading), `phase-0.md` through `phase-11.md`, and `summary.md` (a one-page
 status/sequencing overview of all of them). **Start every session with
 `docs/roadmap/summary.md`** to see where the user is — see "Current phase"
 below. Phase 0 is built test-first from
@@ -34,7 +34,7 @@ It deploys two stacks on a target host via Docker Compose:
 - **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with
  **Grafana Alloy** as the collector and node-exporter for host metrics.
 - **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the
-  hands-on subject of Phase 9, not part of the main pipeline.
+  hands-on subject of Phase 10, not part of the main pipeline.
 Five facts about that platform that change what correct code looks like here. Verify
 them against the live system rather than assuming — they were read from the configs
@@ -91,22 +91,22 @@ These override helpfulness. Breaking any of them ruins the exercise.
 1. **Do not read, summarise, quote, paraphrase, or hint at the contents of
   `docs/PATHOLOGIES.md`** unless the user explicitly says they have finished
-   Phase 8 and wants to review it. Not even "let me check the docs to confirm."
+   Phase 9 and wants to review it. Not even "let me check the docs to confirm."
-2. **Do not reveal which chaos flags are currently enabled** during Phase 8. Do
+2. **Do not reveal which chaos flags are currently enabled** during Phase 9. Do
   not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and
   do not infer the set from the source and announce it. If the user asks you to
   check, remind them what phase they're in and let them insist — then comply.
-3. **Do not run `pytest -m pathology` during Phase 8.** The pathology suite has
+3. **Do not run `pytest -m pathology` during Phase 9.** The pathology suite has
   one named test per flag, so its pass/fail output is a complete answer key.
   Same for `make test-pathology` and `make test-slow`. Running the `contract`,
   `seed`, `baseline` or `guard` markers is fine.
-4. **Do not diagnose the system for the user during Phase 8.** That phase is the
+4. **Do not diagnose the system for the user during Phase 9.** That phase is the
   capstone: they find the bottlenecks from telemetry alone. Your job there is
   to teach *method* (how to read a trace gap, how to reason about saturation)
   and to operate tools they ask you to operate — not to name the culprit. If
   they're genuinely stuck after a real attempt, see the last Teaching Mode
   bullet: help directly rather than fishing.
-5. Outside Phase 8 these relax. Check `docs/roadmap/summary.md` for the current
+5. Outside Phase 9 these relax. Check `docs/roadmap/summary.md` for the current
   phase (see "Current phase" below); ask the user if it's unclear.
 ---
@@ -119,7 +119,7 @@ spoiler rules, Teaching Mode scope, what to build next, or any "where am I /
 what's next" question), **read `docs/roadmap/summary.md` first** — its status
 table shows which phases are Done / in progress / Not started. Then read only
 the `phase-N.md` file for the phase in play (and `intro.md` for platform
-background if needed); don't load all eleven.
+background if needed); don't load all twelve.
 - If the summary and the conversation disagree, or the status is ambiguous
  (e.g. no phase is marked in progress), ask the user before assuming.
@@ -145,10 +145,10 @@ look met:
   lines) in the same change, so the two never disagree. Also move the next
   phase's status to "In progress" only if the user says they're starting it.
-Phase reference: **0a** test suite · **0b** implement to green · 1 concepts +
+Phase reference: **0a** test suite · **0b** implement to green · 1 deployment
-first trace · 2 tracing · 3 logs · 4 metrics · 5 dashboards-as-code ·
+(Compose + Traefik) · 2 concepts + first trace · 3 tracing · 4 logs · 5 metrics ·
-6 k6 + percentiles · 7 SLO/error budget · 8 bottleneck hunt ·
+6 dashboards-as-code · 7 k6 + percentiles · 8 SLO/error budget ·
-9 Filebeat/Logstash/OTel judgment · 10 delivery.
+9 bottleneck hunt · 10 Filebeat/Logstash/OTel judgment · 11 delivery.
 ---
@@ -218,8 +218,8 @@ Decide by what's being touched:
 **Phase 0 is explicitly exempt**, in both halves. The base project was delegated
 to coding agents by design: **0a** writes the tests, **0b** implements against
-them. Build both from their specs without teaching. Phases 1–10 are the learning
+them. Build both from their specs without teaching. **Phase 1 (deployment)** is
-work.
+likewise plumbing — just build it. Phases 2–11 are the learning work.
 If the user says "just do it," "give me the code," or "skip the socratic
 thing" — comply immediately and without negotiation for that work item.
@@ -243,7 +243,7 @@ split collapses if the tests drift toward the implementation.
  counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception
  paths) are deliberately built so faking is harder than implementing; treat that
  as the intent for all of them.
- From Phase 3 onward the log-line-counting assertions in the pathology suite
+- From Phase 4 onward the log-line-counting assertions in the pathology suite
  *should* be rewritten against structured logs and then span/metric data. That
  is a planned migration the user drives — propose it, don't do it unasked, and
  it does not license loosening anything else.
@@ -287,7 +287,7 @@ docker compose logs -f gateway         # actual log shape, not the intended one
 make test-contract                     # fast functional suite (< 60s)
 make test-baseline                     # is the clean system actually healthy
-pytest tests/ -m guard -q              # "no observability yet" — delete at Phase 2
+pytest tests/ -m guard -q              # "no observability yet" — delete at Phase 3
 git diff --stat tests/                 # did anything touch the contract? should be empty
 ```
@@ -379,7 +379,7 @@ Things a well-meaning assistant does wrong here:
 - **Importing application code into `tests/`.** The suite is black-box on
  purpose; an import couples it to implementation and the contract stops meaning
  anything.
- **Writing `common/telemetry.py` for him** in Phase 2. That file is the single
+- **Writing `common/telemetry.py` for him** in Phase 3. That file is the single
  most valuable thing he writes. Checkpoint it: Resource → TracerProvider →
  BatchSpanProcessor → exporter → propagator, one at a time.
 - **Creating or editing dashboards through the Grafana UI or HTTP API** as a
@@ -400,13 +400,13 @@ Things a well-meaning assistant does wrong here:
  Aggregate the histogram buckets, then compute the quantile.
 - **High-cardinality metric labels** — `account_id`, `transaction_id`,
  `merchant_name`, raw paths. These belong on spans. (There is one *deliberate*
-  cardinality-explosion exercise in Phase 4; that one is intentional and gets
+  cardinality-explosion exercise in Phase 5; that one is intentional and gets
  reverted.)
 - **`constant-vus` for load profiles** where an open model is correct.
  Closed-model executors hide latency through coordinated omission.
 - **Putting `trace_id` in a Loki label** instead of the log body + derived field.
 - **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a
-  Phase 9 judgment exercise and a small side pipeline only — not the
+  Phase 10 judgment exercise and a small side pipeline only — not the
  architecture.
 - Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md`
  for him. Those are his reasoning. Review, challenge, poke holes — don't write.
@@ -423,7 +423,7 @@ Things a well-meaning assistant does wrong here:
  `db.system`, `server.address`). Namespace custom ones as `ledgerline.*`.
 - `service.name` matches the directory name under `services/`.
 - Duplication across services is fine. Don't refactor toward a shared framework.
- Docker Compose only, locally, through Phase 9. Kubernetes is a later
+- Docker Compose only, locally, through Phase 10. Kubernetes is a later
  follow-up — porting instrumentation and debugging a cluster at the same time
  is how this stalls.
 - Never save a Grafana dashboard from the UI.
@@ -37,7 +37,7 @@ make test-contract  # fast functional suite (~60s)
 Then open **http://localhost:8000**.
 The observability backend lives in a separate Ansible repo and is deployed to its
-own host. From Phase 1 onward:
+own host. From Phase 2 onward:
 ```bash
 # in observability-ansible/
@@ -88,7 +88,7 @@ Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz`
 (dependency check).
 **Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no
-automatic trace-context propagation, so Phase 2 has to implement it by hand.
+automatic trace-context propagation, so Phase 3 has to implement it by hand.
 That's the most valuable exercise in the project. Don't swap it out.
 ### The primary request flow
@@ -137,7 +137,7 @@ Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`,
 `no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`,
 `chatty_health`.
-> **During Phase 8, don't open the chaos panel and don't call
+> **During Phase 9, don't open the chaos panel and don't call
 > `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the
 > seed; the point is to find the pathologies from telemetry alone. The panel
 > will happily tell you the answer, which is the one thing you don't want.
@@ -160,8 +160,8 @@ Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`,
 | `test-slow` | The soak-ish pathology tests |
 | `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile |
 | `fix-index` / `drop-index` | Add or remove the deliberately missing indexes |
-| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 5+* |
+| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 6+* |
-| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 6+* |
+| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 7+* |
 ---
@@ -198,17 +198,17 @@ make test                 # everything except slow
 ```
 > ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so
-> its output is a complete answer key. Don't run it during the Phase 8
+> its output is a complete answer key. Don't run it during the Phase 9
 > bottleneck hunt.
 `tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no
 `prometheus_client`, no structured logging, no `/metrics`. **Delete it at the
-start of Phase 2**; it has done its job by then.
+start of Phase 3**; it has done its job by then.
 Some pathology assertions currently count log lines, because plain text is the
 only cross-service observability surface that exists at Phase 0. That is the
 deficiency this project sets out to fix — those assertions get rewritten against
-structured logs in Phase 3 and span data in Phase 4, and become both simpler and
+structured logs in Phase 4 and span data in Phase 5, and become both simpler and
 more robust. See `tests/README.md`.
 ---
@@ -321,14 +321,14 @@ Rehearse this before showing anyone.
 └─ docs/
   ├─ roadmap/                       # the ten-phase learning plan, one file per phase
   │  ├─ intro.md                    # platform background, resource spine (read first)
-   │  ├─ phase-0.md … phase-10.md    # one phase per file — status, message, concepts, build
+   │  ├─ phase-0.md … phase-11.md    # one phase per file — status, message, concepts, build
   │  └─ summary.md                  # one-page status/sequencing overview of all phases
   ├─ phase-0a-test-suite-spec.md   # spec the test suite was written from
   ├─ phase-0-base-project-spec.md  # spec the base app was built from
   ├─ PATHOLOGIES.md                # ⚠️ spoilers — the answer key
   ├─ SLO.md                        # SLIs, SLOs, SLA, RTO/RPO, error budget policy
   ├─ CONCEPTS.md                   # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words
-   ├─ INVESTIGATION.md              # the Phase 8 bottleneck hunt, written as an incident doc
+   ├─ INVESTIGATION.md              # the Phase 9 bottleneck hunt, written as an incident doc
   └─ PIPELINE-CHOICES.md           # Filebeat vs Logstash vs OpenTelemetry — when and why
 ```
@@ -338,16 +338,17 @@ Rehearse this before showing anyone.
 - [ ] **Phase 0a** — integration test suite, written first, red against nothing
 - [ ] **Phase 0b** — base project implemented until the suite is green
- [ ] **Phase 1** — concepts, first trace end to end
+- [x] **Phase 1** — deployment: Docker Compose + Traefik on a server
- [ ] **Phase 2** — distributed tracing across all seven services
+- [ ] **Phase 2** — concepts, first trace end to end
- [ ] **Phase 3** — structured logs, correlated with traces
+- [ ] **Phase 3** — distributed tracing across all seven services
- [ ] **Phase 4** — metrics, RED, service map, exemplars
+- [ ] **Phase 4** — structured logs, correlated with traces
- [ ] **Phase 5** — dashboards and alerts as code
+- [ ] **Phase 5** — metrics, RED, service map, exemplars
- [ ] **Phase 6** — k6 load testing, percentiles
+- [ ] **Phase 6** — dashboards and alerts as code
- [ ] **Phase 7** — SLIs, SLOs, error budgets, RTO/RPO
+- [ ] **Phase 7** — k6 load testing, percentiles
- [ ] **Phase 8** — bottleneck hunt
+- [ ] **Phase 8** — SLIs, SLOs, error budgets, RTO/RPO
- [ ] **Phase 9** — Filebeat / Logstash / OpenTelemetry judgment
+- [ ] **Phase 9** — bottleneck hunt
- [ ] **Phase 10** — delivery
+- [ ] **Phase 10** — Filebeat / Logstash / OpenTelemetry judgment
 - [ ] **Phase 11** — delivery
 Full plan: [`docs/roadmap/`](docs/roadmap/) — start with
 [`intro.md`](docs/roadmap/intro.md), then the phase you're on, or
@@ -0,0 +1,63 @@
 # Production-style override: put the gateway (frontend + API) behind Traefik
 # with automatic Let's Encrypt TLS. Use together with the base file:
 #
 #   docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --build
 #
 # Only Traefik publishes host ports (80/443). Everything else is reachable
 # only on the compose network.
 services:
  traefik:
    image: traefik:v3.6
    restart: unless-stopped
    command:
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --certificatesresolvers.le.acme.httpchallenge=true
      - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --log.level=INFO
      - --accesslog=true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - letsencrypt:/letsencrypt
  gateway:
    ports: !reset []
    labels:
      - traefik.enable=true
      - traefik.http.routers.ledgerline.rule=Host(`ledgerline.vione.cloud`)
      - traefik.http.routers.ledgerline.entrypoints=websecure
      - traefik.http.routers.ledgerline.tls.certresolver=le
      - traefik.http.services.ledgerline.loadbalancer.server.port=8000
  postgres:
    ports: !reset []
  redis:
    ports: !reset []
  ledger:
    ports: !reset []
  fraud:
    ports: !reset []
  sanctions:
    ports: !reset []
  insights:
    ports: !reset []
  notifier:
    ports: !reset []
 volumes:
  letsencrypt:
@@ -1,7 +1,7 @@
 # Observability & APM — Delivery Roadmap for Babak's Task
 *Part of `docs/roadmap/` — platform background and reading guidance shared by
-every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`;
+every phase. Per-phase detail lives in `phase-0.md` through `phase-11.md`;
 [`summary.md`](summary.md) gives the one-page overview and status of all of
 them.*
@@ -90,16 +90,16 @@ learn OTel first, every backend becomes "point OTLP at a different URL."
 One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
 early, in parallel with OTel metrics — because it *constrains what you're allowed to
 emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
-(Phases 4–5), not up front.
+(Phases 5–6), not up front.
 Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
-part of the Grafana stack and you do not need them for this task. Phase 9 covers
+part of the Grafana stack and you do not need them for this task. Phase 10 covers
 them as a judgment question, which is exactly how Babak framed them.
 ### Verdict on the OneUptime article you found
 **Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
-teaching resource. Read it in **Phase 4/5**, not now; it assumes you already
+teaching resource. Read it in **Phase 5/6**, not now; it assumes you already
 understand spans, span metrics and PromQL.
 What's genuinely good in it, and rare to find in one place:
@@ -133,7 +133,7 @@ Two caveats:
 This changes *where* your telemetry goes and *how the pipeline is configured*. It
 does not change the learning arc, the app, or the test suite. Read this before
-Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9.
+Phase 2; it rewrites parts of Phases 2, 4, 5, 6, 8 and 10.
 ### What the repo is
@@ -199,7 +199,7 @@ Prometheus`. The shipped config does not do that:
 Neither is wrong; the diagram shows the intended architecture and the code
 implements a simpler one. Closing that gap — adding `prometheus.scrape` +
 `prometheus.remote_write` to Alloy so metrics really do flow through the collector —
-is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth
+is a genuinely good Phase 5 exercise, and **noticing the discrepancy at all is worth
 mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
 exactly the skill this task is meant to build.
@@ -264,7 +264,7 @@ Tempo and remote-written into Prometheus (which runs with
 connectors to Alloy** — you'd be generating the same series twice under different
 names. The metric names differ from the OneUptime article's; go find the actual
 names in Prometheus before writing any PromQL. (`local-blocks` also gives you
-TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.)
+TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.)
 **3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
 datasource is provisioned with:
@@ -295,16 +295,17 @@ your deliverable, and the natural home for it is *inside this Ansible role*.
 | Phase | Impact |
 |---|---|
 | 0 / 0a | **None.** App and test suite are unaffected. |
-| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
+| 1 | **New.** Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes. |
-| 2 | Endpoint changes to Alloy; tracing work itself is identical. |
+| 2 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
-| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
+| 3 | Endpoint changes to Alloy; tracing work itself is identical. |
-| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
+| 4 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
-| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
+| 5 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
-| 6 | None. |
+| 6 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
-| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
+| 7 | None. |
-| 8 | None, plus TraceQL metrics from `local-blocks`. |
+| 8 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
-| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
+| 9 | None, plus TraceQL metrics from `local-blocks`. |
-| 10 | Demo now includes `ansible-playbook` rebuilding the platform. |
+| 10 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
 | 11 | Demo now includes `ansible-playbook` rebuilding the platform. |
 ### Where to run it
@@ -326,7 +327,7 @@ Is this repo a **platform you must consume as-is**, or a **starting point you ma
 modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
 assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
 extend `group_vars`. If Babak intends it as a fixed platform, several
-recommendations below change — ask before Phase 3.
+recommendations below change — ask before Phase 4.
 ---
@@ -340,8 +341,8 @@ subtitle-to-notes workflow.
 | Resource | Format | Length | Cost | Best for |
 |---|---|---|---|---|
-| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. |
+| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 4–6. Comes with browser-based Killercoda playgrounds. |
-| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. |
+| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 3–5. |
 If budget allows, buy both — they overlap by maybe 30% and the second half of each
 is different. If only one: take the **OpenTelemetry** one, because instrumentation
@@ -351,27 +352,27 @@ is the part you have to *write*, and Grafana's own docs are excellent for the re
 | Resource | Format | Length | Use in |
 |---|---|---|---|
-| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
+| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 2. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
-| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. |
+| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 3. The best Python-specific overview, with a companion repo. |
-| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. |
+| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 3. Auto- vs manual-instrumentation side by side. |
 | **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
-| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
+| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 2. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
-| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. |
+| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 2–5. The neutral source of truth. |
-| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 |
+| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 6–7 |
-| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
+| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 2–5. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
-| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. |
+| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 5, 9. This is where your RED metrics and service graph actually come from. |
-| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on |
+| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 10, now hands-on |
-| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 |
+| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 8 |
-| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. |
+| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 8. Self-run the exercises with Ledgerline as the subject service. |
-| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
+| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 8, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
-| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 |
+| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 9 |
-| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. |
+| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 5. The clearest explanation anywhere of what cardinality actually costs. |
-| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 |
+| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 3 |
-| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 |
+| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 10 |
-**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on
+**Where to run it:** Docker Compose on your laptop for Phases 0–9. Don't put this on
 your k8s cluster until everything works locally — debugging instrumentation and
 debugging Kubernetes at the same time is how these projects die. Porting it to the
-cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half
+cluster is a good Phase 11+ follow-up, and it's the natural bridge to the DevOps half
 Babak has planned for you.
@@ -67,7 +67,7 @@ deliberate brokenness.
 ### Why TDD here, when the app itself doesn't matter
-The microservices are throwaway. The *properties* aren't. Everything from Phase 2
+The microservices are throwaway. The *properties* aren't. Everything from Phase 3
 onward assumes the system still behaves a specific way: seven real services in
 the path, async work genuinely off the request path, five hot accounts owning a
 third of the rows, and thirteen pathologies that actually degrade something. You
@@ -114,13 +114,13 @@ Done when `make test` is green three consecutive runs and
 ### Then, yourself
 1. **Read every service's `app.py` end to end.** You need to know this codebase
-   cold, because from Phase 2 onward you're modifying all seven services.
+   cold, because from Phase 3 onward you're modifying all seven services.
 2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
   then try to figure out what's wrong using only `docker compose logs`.** Give it
   twenty real minutes. You will fail, and the specific way you fail — no timing,
   no causality, no cross-service correlation, a `X-Request-Id` that dies at the
   Redis Stream — is the motivation for everything that follows. Write down what
-   you wished you had. That list is your acceptance criteria for Phase 8.
+   you wished you had. That list is your acceptance criteria for Phase 9.
 **Done when:** `make test` is green three runs in a row, all nine acceptance
 criteria in the base spec pass, and you have that written list of frustrations.
@@ -1,89 +1,74 @@
-# Phase 1 — The model, and one signal end to end
+# Phase 1 — Deployment: Ledgerline on a server behind Traefik
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
-**Status:** Not started
+**Status:** Done
-**Message:** Blocked on nothing — Phase 0 is done and the platform is
+**Message:** Ledgerline is running on a remote server and served over HTTPS at
-described above. Waiting to begin.
+**https://ledgerline.vione.cloud**. Verified against the live deployment, not
 just the config: the frontend loads, `/api/health/all` reports every service
 `ok`, and the origin serves a real Let's Encrypt certificate (valid until
 2027-01-07, auto-renewed by Traefik).
-**Time-box:** one evening.
+- **Target:** the throwaway server `czy` (Ubuntu 24.04, 1 vCPU, 1 GB RAM), repo
  at `/opt/apps/LedgerLine`. Docker 29.9 and Compose v5.6 installed from
  Docker's apt repo. A 2 GB swapfile was added (persisted in `/etc/fstab`) —
  nine containers do not fit in 1 GB of RAM without it, and steady state still
  uses roughly 700 MB plus swap.
 - **How it's wired:** `docker-compose.traefik.yml` is an *override*, not an
  edit of `docker-compose.yml`, so local development is unchanged. It adds
  Traefik v3.6 on ports 80/443 (Docker provider, `exposedByDefault=false`,
  HTTP→HTTPS redirect, Let's Encrypt via HTTP-01), routes
  ``Host(`ledgerline.vione.cloud`)`` to the gateway on port 8000, and strips
  the published host ports from every other service (`ports: !reset []`) so
  Postgres, Redis and the services are reachable only on the Compose network.
 - **No application or `tests/` changes.** The Dockerfile and base compose file
  are untouched; schema applied and data seeded on the server with the same
  steps as `make seed` (800 accounts, 250,000 transactions).
 - **Deploy / redeploy command** (always both files — a bare `docker compose up`
  would use the base file alone and publish every port):
-**Why:** you need the vocabulary, but more importantly you need to see one trace
+  ```bash
-appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
+  docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --build
-than abstract.
+  ```
 - **Cloudflare gotchas hit on the way:**
  - The first certificate attempt failed with an ACME 404 because the
    Cloudflare DNS record did not yet point at the server — a packet capture
    on the server showed no inbound traffic from Cloudflare at all. Traefik
    only retries issuance on restart, so after fixing the record it needed
    `docker compose ... restart traefik`.
  - With the orange-cloud proxy on, the SSL/TLS mode must be **Full** —
    **Flexible** combined with Traefik's HTTPS redirect causes a redirect loop.
 - **Caveats to remember:** the deployment is public, including the
  `/api/admin/chaos` endpoints (anyone who finds them can flip flags);
  internet scanners probed the site within seconds of it going live. The
  server is disposable by design, and Traefik's state lives in the
  `letsencrypt` volume, which is lost on an OS reinstall (a new certificate is
  simply issued on the next run).
 **Time-box:** 1 evening.
 **Deliverable:** the unmodified-behaviour Ledgerline system reachable at a real
 domain over HTTPS, reproducible with one compose command.
 ### Concepts
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
+- **Traefik's model:** *entrypoints* (the ports it listens on), *routers*
-  (known unknowns): dashboards and alerts built from predefined aggregates.
+  (rule → service), *services* (where to send traffic), *middlewares*, and
-  Observability is a property of the system: can you ask a *new* question about
+  *providers* (where config comes from — here, Docker labels on containers).
-  production, that you never anticipated, without shipping new code? Charity Majors'
+  Compare with nginx, where the same information lives in one static config
-  framing — monitoring is known-unknowns, observability is unknown-unknowns. The
+  file.
-  practical implication for you: emit **wide, high-dimensional events** (rich span
+- **Compose override files.** `-f base -f override` merges the files;
-  attributes), not just pre-aggregated counters.
+  `!reset` clears an inherited key (here, `ports`) instead of appending to it.
- **The OTel architecture split: API vs SDK vs Collector.**
+- **ACME HTTP-01 behind a CDN.** Let's Encrypt must be able to reach port 80 on
-  - The **API** is what your code and third-party instrumentation libraries call.
+  the *hostname's actual origin*. With Cloudflare proxying, that only works
-    If no SDK is configured, API calls are no-ops — that's deliberate, it means a
+  once the DNS record points at the right server.
-    library can be instrumented without forcing a dependency on any implementation.
+- **Why this phase exists before observability.** The telemetry work in
-  - The **SDK** is the engine: sampling, batching, context propagation, export.
+  Phases 2–11 is more convincing when the system under observation is a real
-    Application owners configure the SDK; library authors only touch the API.
+  deployment rather than a laptop, and the platform (`observability-ansible`)
-  - The **Collector** is a standalone process that receives, processes and exports
+  is already a remote deployment — Ledgerline now sits on the same footing.
    telemetry. Optional, but you should use one.
 - **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
 - **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
  `service.version`, `deployment.environment`) and is attached to every signal from
  that process. Attributes describe the individual span/metric/log.
 - **Semantic conventions**: standardised attribute names (`http.request.method`,
  `db.system`, `server.address`). Use them for anything standard; namespace your own
  (`ledgerline.account.id`). This is what makes cross-service, cross-language
  correlation work — and it's what Grafana's prebuilt dashboards key off.
 - **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
  aware of, not learning yet.)
 - **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
  app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
  seven services. The Collector also does the things you don't want in your app:
  retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
  destinations.
 ### Resources
 - freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
  M.E.L.T. and context-propagation sections; skip the Node code.
 - OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
 - Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
  repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
  it — just see what "done" looks like.
 ### Build
 1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
   `ansible-galaxy collection install -r requirements.yml`, change
   `grafana_admin_password` (vault it), then
   `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
   README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
   Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
   Then open the Alloy UI and **read the component graph** — it shows you the exact
   pipeline, including the fact that the OTLP receiver has only a traces output.
   Seeing that in the UI before you get bitten by it is worth ten minutes.
   *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
   a single container with everything pre-wired. Useful for a first trace, but
   don't build on it — its pipeline differs from your real platform in exactly the
   ways that matter.)*
 2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
   auto-instrumentation:
   ```
   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
   ```
   and change its command to
   `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
   with `OTEL_SERVICE_NAME=sanctions`,
   `OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
   `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
   Tempo directly.
 3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
 **Done when:** you can see a single-service trace in Tempo, and you can explain out
 loud what the Resource attributes on it mean and where they came from.
 **Done when:** `https://ledgerline.vione.cloud` shows the frontend with a valid
 certificate; `/api/health/all` is all `ok`; only ports 22, 80 and 443 are
 reachable on the server; and the deploy is one documented compose command.
@@ -1,40 +1,161 @@
-# Phase 10 — Package the delivery
+# Phase 10 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
-*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)*
+*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`phase-11.md`](phase-11.md)*
 **Status:** Not started
-**Message:** Depends on every prior phase being done — this is the wrap-up,
+**Message:** Not technically blocked by earlier phases (it stands on its own
-not independent work.
+as a judgment exercise), but sequenced last-but-one so Phase 9's incident to
 re-diagnose in Kibana already exists.
-**Time-box:** one evening.
+**Time-box:** 1–2 evenings.
-Babak gave you a task; deliver it like an engineer, not like a student.
+Babak's remaining questions are all *judgment* questions. He isn't testing whether
 you can recite what Filebeat does; he's testing whether you have a framework for
 choosing. Here's the framework, then the answers as worked examples.
-1. **`README.md`** — architecture diagram, one-command start, what each container is
+### The framework — six axes to reason on
-   for, and a "here's the demo" script.
+
-2. **The five-minute demo path**, rehearsed:
+Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
-   - Delete every dashboard in the Grafana UI, then `ansible-playbook
+
-     playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
+1. **Where does the data originate?** Inside your application (you can instrument it)
-     from code, one command.
+   or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
-   - Submit one transaction from the frontend, then follow it through Grafana: RED
+   touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
-     dashboard → latency spike → exemplar → trace → span → logs for that span → back
+2. **What shape is it already in?** Structured JSON at the source, or unstructured
-     to the trace. One request, all three signals, no searching.
+   text needing grok/regex parsing? Parsing cost has to land *somewhere*.
-   - `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
+3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
-     and the service map turn red at a specific node.
+4. **What's the destination, and how many?** One backend forever, or several, or one
-   - Show `docs/INVESTIGATION.md` and how you found the pathologies.
+   you expect to change?
-3. **The written answers** Babak asked for, as documents in the repo, not as a
+5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
-   conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
+   on every host) distributes cost and adds resilience but multiplies config surface.
-   `docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
+   Parsing centrally concentrates cost and creates a single point of failure — but
-   RTO/RPO/SLA/SLO/error budget in your own words.
+   one place to change rules.
-4. **The test suite as part of the story.** Show `make test` green, then show the
+6. **What already exists, and what does it cost?** Existing Elastic investment,
-   three generations of the call-counting assertion side by side —
+   existing licences, team skills, and licensing constraints (Elastic's Filebeat and
-   `docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
+   Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
-   each signal bought you. Most people demo dashboards; almost nobody can show
+   Bit and Vector are permissively licensed). Migration is never free.
-   their observability work paying off inside their own test suite.
+
-5. **A "what I'd do differently in production" section.** Tail sampling instead of
+### The answers
-   100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
+
-   log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
+**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
-   you *didn't* do, and why, is the strongest signal of seniority in the whole
+from a host or container and forwards them with minimal transformation — and your
-   delivery.
+destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
 delivery with a registry of file offsets, backpressure handling, and prebuilt modules
 for common services (nginx, MySQL, system logs) that come with parsing and dashboards
 included. Reach for it when the source is a file you don't control the format of, the
 app can't be modified, and Elasticsearch is where it's going.
 **When to use Logstash.** When you need real *transformation*: parsing unstructured
 text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
 splitting or merging events, conditional routing to multiple destinations, or
 buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
 for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
 per host.
 **When to use Filebeat and Logstash together, and why.** This is the classic Elastic
 architecture and the reason is a **division of labour between edge and centre**:
 - Filebeat runs on *every* host: cheap, tails files, handles offsets and
  backpressure, ships onward. You want the thing running in a thousand places to be
  small and boring.
 - Logstash runs on a *few* central nodes: does the expensive grok parsing,
  enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
  you want them in a handful of places you can redeploy, not on a thousand hosts.
 - It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
  persistent queue absorbs it and Filebeat backs off gracefully, rather than every
  host queuing independently or dropping.
 So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
 Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
 need central buffering. Skip Logstash when your logs are already structured JSON and
 Elasticsearch ingest pipelines can do the little parsing that remains.
 **When to use OpenTelemetry.**
 - You own the application code and want telemetry generated *from inside it* — spans
  with business context, custom metrics, logs already carrying trace IDs.
 - You need **more than one signal**, correlated. This is the strongest argument: one
  SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
  threaded through all three. Neither Filebeat nor Logstash does traces.
 - You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
  data model across all of them.
 - You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
  Jaeger by editing Collector YAML.
 - You need distributed tracing at all. There is no serious vendor-neutral alternative.
 - Your telemetry needs processing in flight — redaction, tail sampling, attribute
  rewriting — without touching the app.
 **When *not* to use OpenTelemetry.** This is the question that separates people who
 have used it from people who have read about it.
 - **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
  an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
  file-based collector.
 - **You only need one signal from one source with no correlation.** A single nginx
  access log going to a single log store: OTel is more machinery than the job needs.
  Fluent Bit or Filebeat is simpler and finishes the job today.
 - **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
  Elastic estate with hundreds of Beats modules, dashboards and detection rules is
  years of investment. "We should use OTel because it's the standard" is not a
  business case. Note the direction of travel though: the Collector can already
  receive from and export to most of these, so a *gradual* migration is usually the
  right shape — often by putting a Collector or Vector in front of the existing
  agents rather than ripping them out.
 - **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
  where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
 - **The signal is genuinely business analytics, not telemetry.** Clickstream data for
  a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
  retention, different query patterns, different cost model.
 - **Maturity gaps for a specific need in a specific language.** OTel logs support has
  matured at different rates per language, and profiling is still alpha. Check the
  status page for your language before betting on a signal.
 - **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
  on it. A two-person team that needs error tracking by Friday should install Sentry.
 The senior framing to give Babak: *these tools aren't competitors so much as
 different layers.* OpenTelemetry is a **generation and transport standard** for
 telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
 don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
 Elasticsearch are **stores**. Most real architectures use one from each layer, and
 the interesting question is always which layer a given problem lives in.
 ### Resources
 - Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
 - SigNoz — *OpenTelemetry vs ELK Stack* (free article).
 - OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
  read these to see how the Collector does the Filebeat/Logstash job itself.
 - OpenTelemetry docs → *Instrumentation status by language* — for the maturity
  caveats.
 ### Build — and this is no longer hypothetical
 Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
 deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
 Filebeat reading Docker container logs and Logstash parsing JSON into daily
 `logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
 questions — the codebase he handed you contains both worlds.
 1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
   flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
   Alloy → Loki. Compare them on the same data.
 2. Test the direct ingest path the README documents — pipe a JSON line into
   Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
   parsing happens before you change it.
 3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
   that parses them. Then do the same job with Alloy's `loki.process` stages. Time
   both. Note which one you'd rather debug at 2 a.m.
 4. Compare the *investigation experience* directly: take one incident you already
   diagnosed in Phase 9 and re-run it in Kibana Discover. What can you do there that
   you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
   wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
   for some questions, and Loki's cost model genuinely beats Elasticsearch's for
   others.
 5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
   licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
   permissively licensed. For some employers that alone decides it.
 6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
   valuably — what you actually found annoying about each. Having run both stacks
   against the same application is the difference between reciting a comparison and
   arguing one.
@@ -0,0 +1,40 @@
 # Phase 11 — Package the delivery
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-10.md`](phase-10.md) · Next: [`summary.md`](summary.md)*
 **Status:** Not started
 **Message:** Depends on every prior phase being done — this is the wrap-up,
 not independent work.
 **Time-box:** one evening.
 Babak gave you a task; deliver it like an engineer, not like a student.
 1. **`README.md`** — architecture diagram, one-command start, what each container is
   for, and a "here's the demo" script.
 2. **The five-minute demo path**, rehearsed:
   - Delete every dashboard in the Grafana UI, then `ansible-playbook
     playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
     from code, one command.
   - Submit one transaction from the frontend, then follow it through Grafana: RED
     dashboard → latency spike → exemplar → trace → span → logs for that span → back
     to the trace. One request, all three signals, no searching.
   - `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
     and the service map turn red at a specific node.
   - Show `docs/INVESTIGATION.md` and how you found the pathologies.
 3. **The written answers** Babak asked for, as documents in the repo, not as a
   conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
   `docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
   RTO/RPO/SLA/SLO/error budget in your own words.
 4. **The test suite as part of the story.** Show `make test` green, then show the
   three generations of the call-counting assertion side by side —
   `docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
   each signal bought you. Most people demo dashboards; almost nobody can show
   their observability work paying off inside their own test suite.
 5. **A "what I'd do differently in production" section.** Tail sampling instead of
   100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
   log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
   you *didn't* do, and why, is the strongest signal of seniority in the whole
   delivery.
@@ -1,116 +1,89 @@
-# Phase 2 — Distributed tracing
+# Phase 2 — The model, and one signal end to end
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
 **Status:** Not started
-**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
+**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
 described above. Waiting to begin.
-**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
+**Time-box:** one evening.
-**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
+**Why:** you need the vocabulary, but more importantly you need to see one trace
-(Resource, providers, exporters, Collector wiring) is shared across all three
+appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
-signals — do it once here and logs and metrics become incremental. Second, tracing
+than abstract.
 is what Babak actually asked for ("trace a single request, see where it stuck"),
 and once trace IDs exist, log correlation in Phase 3 works on the first try instead
 of needing rework.
 ### Concepts
- **Trace / span / span context.** A trace is one request's journey; a span is one
+- **Monitoring vs observability.** Monitoring answers questions you knew to ask
-  timed operation within it. Each span carries a trace ID, its own span ID, and its
+  (known unknowns): dashboards and alerts built from predefined aggregates.
-  parent's span ID. That parent pointer is the entire data structure.
+  Observability is a property of the system: can you ask a *new* question about
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
+  production, that you never anticipated, without shipping new code? Charity Majors'
-  outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
+  framing — monitoring is known-unknowns, observability is unknown-unknowns. The
-  these right is what makes the service graph render correctly.
+  practical implication for you: emit **wide, high-dimensional events** (rich span
- **Context propagation.**
+  attributes), not just pre-aggregated counters.
-  - *In-process*: OTel uses Python `contextvars`. This works across `await` — but
+- **The OTel architecture split: API vs SDK vs Collector.**
-    **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
+  - The **API** is what your code and third-party instrumentation libraries call.
-    carry the context explicitly.
+    If no SDK is configured, API calls are no-ops — that's deliberate, it means a
-  - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
+    library can be instrumented without forcing a dependency on any implementation.
-    `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
+  - The **SDK** is the engine: sampling, batching, context propagation, export.
-  - The classic bug: any boundary that doesn't carry those headers **starts a brand
+    Application owners configure the SDK; library authors only touch the API.
-    new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
+  - The **Collector** is a standalone process that receives, processes and exports
-    `TraceContextTextMapPropagator().inject(...)` into the message payload on the
+    telemetry. Optional, but you should use one.
-    producer and `.extract(...)` on the consumer, then starting the consumer span
+- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
-    with that context as parent. This is the single most valuable exercise in the
+- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
-    phase — most people never do it manually and never really understand it.
+  `service.version`, `deployment.environment`) and is attached to every signal from
-  - Related: `links` (for a consumer span that batches messages from many traces)
+  that process. Attributes describe the individual span/metric/log.
-    and **Baggage** (arbitrary key/values propagated alongside trace context —
+- **Semantic conventions**: standardised attribute names (`http.request.method`,
-    useful, and a foot-gun, because it's sent on every hop and is visible to
+  `db.system`, `server.address`). Use them for anything standard; namespace your own
-    downstream systems; never put secrets in it).
+  (`ledgerline.account.id`). This is what makes cross-service, cross-language
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
+  correlation work — and it's what Grafana's prebuilt dashboards key off.
-  client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
+- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
-  gets you the business semantics: `score_transaction`, `check_sanctions_list`,
+  aware of, not learning yet.)
-  `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
+- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
-  that show *which service* was slow but not *why*.
+  app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
- **Span attributes vs metric labels.** Spans are the right home for
+  seven services. The Collector also does the things you don't want in your app:
-  high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
+  retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
-  with volume, not with distinct values. This is why "put it on the span, not the
+  destinations.
  metric" is the standing rule.
 - **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
  That's how the exception ends up attached to the trace and how the `spanmetrics`
  connector in Phase 4 will count your error rate.
 - **Sampling — the concept now, the config later.**
  - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
    errors at the same rate as successes.
  - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
    traces, but must run in the Collector and buffer spans in memory.
  - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
    now; you'll configure tail sampling in the Collector in Phase 5.
  - Whatever you choose, the sampling decision must **propagate** — the sampled flag
    rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
    unilaterally drop half of a trace its caller decided to keep.
 ### Resources
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
+- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
-  min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
+  M.E.L.T. and context-propagation sections; skip the Node code.
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
+- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
+- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
-  practical FastAPI-specific setup, including batch/queue env vars.
+  repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
+  it — just see what "done" looks like.
  reading.
 - **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
  side by side. Look specifically at how they mix auto-instrumentation with
  hand-rolled business spans, and how the Kafka producer/consumer propagate context.
  This is the "professional implementation" you asked to study.
 ### Build
-1. **Shared bootstrap.** Create `common/telemetry.py` with a single
+1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
-   `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
+   `ansible-galaxy collection install -r requirements.yml`, change
-   `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
+   `grafana_admin_password` (vault it), then
-   service calls it once at startup. Keep it under 60 lines.
+   `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
-2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
+   README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
-   `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
+   Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
-   you now get a connected trace across `gateway → ledger → fraud → sanctions`.
+   Then open the Alloy UI and **read the component graph** — it shows you the exact
-3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
+   pipeline, including the fact that the OTLP receiver has only a traces output.
-   `sanctions.lookup_watchlist`, `insights.aggregate_120d`,
+   Seeing that in the UI before you get bitten by it is worth ten minutes.
-   `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
+   *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
-   namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
+   a single container with everything pre-wired. Useful for a first trace, but
-   `ledgerline.tx.amount_minor`).
+   don't build on it — its pipeline differs from your real platform in exactly the
-4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
+   ways that matter.)*
-   inject trace context into the message fields. Consumers (`notifier`, `audit`):
+2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
-   extract, and start a `CONSUMER` span with the extracted context as parent.
+   auto-instrumentation:
-5. **Record exceptions** everywhere you currently `logger.exception(...)`.
+   ```
-6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
+   pip install opentelemetry-distro opentelemetry-exporter-otlp
-   more.
+   opentelemetry-bootstrap -a install
-7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
+   ```
-   the "before" state so you could measure against it, and from here it's
+   and change its command to
-   actively wrong. Everything else in `tests/` must still pass: instrumenting
+   `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
-   the system is not allowed to change its behaviour, and if `make test` goes red
+   with `OTEL_SERVICE_NAME=sanctions`,
-   after adding spans, you've broken something real. That is exactly the safety
+   `OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
-   net you built it for.
+   `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
   Tempo directly.
 3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
-### Done when
+**Done when:** you can see a single-service trace in Tempo, and you can explain out
-
+loud what the Resource attributes on it mean and where they came from.
 - One request produces **one connected trace** covering all seven services,
  including both async consumers.
 - With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
  ate the time.
 - You can explain why `blocking_event_loop` looks different in a trace than
  `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
  before spans start, plus every concurrent request in that process degrading
  together.)
@@ -1,99 +1,116 @@
-# Phase 3 — Structured logs, correlated to traces
+# Phase 3 — Distributed tracing
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
 **Status:** Not started
-**Message:** Depends on Phase 2 (trace IDs must exist before log correlation
+**Message:** Depends on Phase 2's `common/telemetry.py` bootstrap existing first.
 means anything).
-**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
+**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
 **Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
 (Resource, providers, exporters, Collector wiring) is shared across all three
 signals — do it once here and logs and metrics become incremental. Second, tracing
 is what Babak actually asked for ("trace a single request, see where it stuck"),
 and once trace IDs exist, log correlation in Phase 4 works on the first try instead
 of needing rework.
 ### Concepts
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
+- **Trace / span / span context.** A trace is one request's journey; a span is one
-  events: a stable event name plus typed fields (`event="transaction_settled"`,
+  timed operation within it. Each span carries a trace ID, its own span ID, and its
-  `account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
+  parent's span ID. That parent pointer is the entire data structure.
-  Consistent field names across all seven services — `account_id` everywhere, never
+- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
-  `accountId` in one place and `acct` in another. This is what makes LogQL queries
+  outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
-  possible at all.
+  these right is what makes the service graph render correctly.
- **Trace–log correlation.** Every log line emitted inside an active span must carry
+- **Context propagation.**
-  `trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
+  - *In-process*: OTel uses Python `contextvars`. This works across `await` — but
-  "one pane of glass": click a span → see its logs; click a log line → open its
+    **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
-  trace. Both directions are configured, and you'll set both up.
+    carry the context explicitly.
- **Two ways to ship logs, and which one your platform actually does:**
+  - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
-  - **stdout → collector scraping → Loki.** The container-native way. Your app just
+    `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
-    prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
+  - The classic bug: any boundary that doesn't carry those headers **starts a brand
-    standard in Kubernetes. **This is the only path wired in your Alloy config**
+    new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
-    (`loki.source.docker` reading the Docker socket).
+    `TraceContextTextMapPropagator().inject(...)` into the message payload on the
-  - **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
+    producer and `.extract(...)` on the consumer, then starting the consumer span
-    with the same Resource attributes attached automatically. Cleaner correlation,
+    with that context as parent. This is the single most valuable exercise in the
-    but the app owns delivery. **On your platform this currently goes nowhere** —
+    phase — most people never do it manually and never really understand it.
-    Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
+  - Related: `links` (for a consumer span that batches messages from many traces)
-    dropped in silence.
+    and **Baggage** (arbitrary key/values propagated alongside trace context —
-  - Do both anyway: use stdout for most services, and *extend Alloy* to wire the
+    useful, and a foot-gun, because it's sent on every hop and is visible to
-    OTLP logs path for one or two. Then argue the trade-off from experience. That
+    downstream systems; never put secrets in it).
-    comparison — including the fact that a misconfigured pipeline fails silently
+- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
-    rather than loudly — is precisely what Babak will probe.
+  client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
- **The field name is not a style choice here.** Your Loki datasource is provisioned
+  gets you the business semantics: `score_transaction`, `check_sanctions_list`,
-  with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
+  `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
-  A log line must carry lowercase `trace_id`. OpenTelemetry's logging
+  that show *which service* was slow but not *why*.
-  auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
+- **Span attributes vs metric labels.** Spans are the right home for
-  logs and traces both arriving, no errors anywhere, and no clickable link. Hence
+  high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
-  structlog with an explicit processor rather than the auto-injection shortcut.
+  with volume, not with distinct values. This is why "put it on the span, not the
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
+  metric" is the standing rule.
-  your entire fleet, and Loki bills/indexes on volume.
+- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
+  That's how the exception ends up attached to the trace and how the `spanmetrics`
-  `json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
+  connector in Phase 5 will count your error rate.
-  redaction into the processor chain, not into each call site.
+- **Sampling — the concept now, the config later.**
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
+  - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
-  not log content. Labels should be few and low-cardinality (`service`, `env`,
+    errors at the same rate as successes.
-  `level`). Everything else lives in the log line body and is filtered at query time.
+  - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
-  Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
+    traces, but must run in the Collector and buffer spans in memory.
-  in the JSON body and use a **derived field** to make it clickable.
+  - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
    now; you'll configure tail sampling in the Collector in Phase 6.
  - Whatever you choose, the sampling decision must **propagate** — the sampled flag
    rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
    unilaterally drop half of a trace its caller decided to keep.
 ### Resources
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
+- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
-  Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
+  min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
-  setup, including the hex formatting for trace/span IDs.
+- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
+- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
-  `OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
+  practical FastAPI-specific setup, including batch/queue env vars.
-  fields, if you'd rather not hand-roll a processor.
+- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
- Udemy backbone: the Loki + LogQL sections.
+  reading.
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
+- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
  side by side. Look specifically at how they mix auto-instrumentation with
  hand-rolled business spans, and how the Kafka producer/consumer propagate context.
  This is the "professional implementation" you asked to study.
 ### Build
-1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
+1. **Shared bootstrap.** Create `common/telemetry.py` with a single
-   processor that pulls the current span context and injects `trace_id` / `span_id`.
+   `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
-2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
+   `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
-   handler: request received (or rely on the span), business decision, error.
+   service calls it once at startup. Keep it under 60 lines.
-3. Add a redaction processor with a denylist of field names.
+2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
-4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
+   `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
-   with a `container` label (Alloy's relabel rule sets it from the Docker container
+   you now get a connected trace across `gateway → ledger → fraud → sanctions`.
-   name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
+3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
-5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
+   `sanctions.lookup_watchlist`, `insights.aggregate_120d`,
-   send some records, and watch nothing appear in Loki. Then check the Alloy UI:
+   `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
-   the receiver shows traffic, and the pipeline shows no logs output. Now extend
+   namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
-   `config.alloy.j2` — add a `logs` output on the receiver plus an
+   `ledgerline.tx.amount_minor`).
-   `otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
+4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
-   them arrive. Write down what you'd have done to debug this if you *hadn't* known.
+   inject trace context into the message fields. Consumers (`notifier`, `audit`):
-   That is the most transferable half-hour in the whole project.
+   extract, and start a `CONSUMER` span with the extracted context as parent.
-6. Confirm the provisioned correlation works in both directions: Loki's
+5. **Record exceptions** everywhere you currently `logger.exception(...)`.
-   `derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
+6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
-   (`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
+   more.
-   are verifying, not writing. If a link doesn't appear, the regex didn't match
+7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
-   your field name — check that before anything else.
+   the "before" state so you could measure against it, and from here it's
-6. **Migrate the test suite's log-counting helpers.** Three pathology tests
+   actively wrong. Everything else in `tests/` must still pass: instrumenting
-   (`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
+   the system is not allowed to change its behaviour, and if `make test` goes red
-   in `docker compose logs` — the only cross-service call counter that existed at
+   after adding spans, you've broken something real. That is exactly the safety
-   Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
+   net you built it for.
   with LogQL over structured fields instead. This is a small change and a
   pointed one: notice how much shorter and more precise the assertion gets, and
   that it now works without shelling out to Docker at all. That delta is the
   value of structured logging, measured on your own code.
-**Done when:** from a slow trace you can click into that request's logs, and from
+### Done when
-any log line you can click into its trace — in both directions, in two clicks.
+
 - One request produces **one connected trace** covering all seven services,
  including both async consumers.
 - With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
  ate the time.
 - You can explain why `blocking_event_loop` looks different in a trace than
  `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
  before spans start, plus every concurrent request in that process degrading
  together.)
@@ -1,126 +1,99 @@
-# Phase 4 — Metrics
+# Phase 4 — Structured logs, correlated to traces
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
 **Status:** Not started
-**Message:** Not blocked technically (metrics don't strictly need tracing
+**Message:** Depends on Phase 3 (trace IDs must exist before log correlation
-first), but sequenced after Phase 3 per the roadmap's ordering.
+means anything).
-**Time-box:** 2 evenings.
+**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
 ### Concepts
- **Instrument types.**
+- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
-  - **Counter** — monotonic, only increases. Requests, errors, messages consumed.
+  events: a stable event name plus typed fields (`event="transaction_settled"`,
-  - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
+  `account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
-    connection-pool checkouts.
+  Consistent field names across all seven services — `account_id` everywhere, never
-  - **Histogram** — a distribution. Request latency, payload size. This is what
+  `accountId` in one place and `acct` in another. This is what makes LogQL queries
-    gives you percentiles.
+  possible at all.
-  - Synchronous instruments are recorded inline; **observable/asynchronous** ones are
+- **Trace–log correlation.** Every log line emitted inside an active span must carry
-    read via a callback on collection (right for things you *poll*, like pool size).
+  `trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
-  - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
+  "one pane of glass": click a span → see its logs; click a log line → open its
-    **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
+  trace. Both directions are configured, and you'll set both up.
-    nonsense. Use histograms.
+- **Two ways to ship logs, and which one your platform actually does:**
- **Cardinality — the one thing that will bite you.** Every unique combination of
+  - **stdout → collector scraping → Loki.** The container-native way. Your app just
-  metric name + label values is a separate time series in memory. Add `account_id`
+    prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
-  as a label on a 800-account demo and you've made 800 series; do it in production
+    standard in Kubernetes. **This is the only path wired in your Alloy config**
-  and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
+    (`loki.source.docker` reading the Docker socket).
-  ~10 000 distinct values in production, it does not belong on a metric.** Put it on
+  - **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
-  a span. Histograms multiply this (one series per bucket per label combination), so
+    with the same Resource attributes attached automatically. Cleaner correlation,
-  don't over-bucket either.
+    but the app owns delivery. **On your platform this currently goes nowhere** —
-  - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
+    Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
-    the concrete path), `http.status_code` class, `kind`, `currency`.
+    dropped in silence.
-  - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
+  - Do both anyway: use stdout for most services, and *extend Alloy* to wire the
-    raw URL, timestamp, error message.
+    OTLP logs path for one or two. Then argue the trade-off from experience. That
- **RED and USE.**
+    comparison — including the fact that a misconfigured pipeline fails silently
-  - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
+    rather than loudly — is precisely what Babak will probe.
-    per-service dashboard.
+- **The field name is not a style choice here.** Your Loki datasource is provisioned
-  - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
+  with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
-    connection pools, queues, disk. Saturation is the one everyone forgets and the
+  A log line must carry lowercase `trace_id`. OpenTelemetry's logging
-    one that predicts collapse.
+  auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
-  - You want both: RED tells you the service is slow, USE tells you which resource
+  logs and traces both arriving, no errors anywhere, and no clickable link. Hence
-    is why.
+  structlog with an explicit processor rather than the auto-injection shortcut.
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
+- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
-  and knowing both is the point:
+  your entire fleet, and Loki bills/indexes on volume.
-  - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
+- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
-    `servicegraph` connectors read spans in flight and emit RED metrics and edge
+  `json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
-    metrics. This is what most blog posts (including the OneUptime article) show.
+  redaction into the processor chain, not into each call site.
-  - *In the trace store:* Tempo's **metrics-generator** does the same job after
+- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
-    ingestion, remote-writing the result to Prometheus. **This is what your platform
+  not log content. Labels should be few and low-cardinality (`service`, `env`,
-    does** — `processors: [service-graphs, span-metrics, local-blocks]` with
+  `level`). Everything else lives in the log line body and is filtered at query time.
-    `send_exemplars: true`, remote-writing to a Prometheus started with
+  Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
-    `--web.enable-remote-write-receiver`.
+  in the JSON body and use a **derived field** to make it clickable.
  - So RED metrics, the service map and exemplars **already exist** before you write
    a single metric line. Do not add the collector connectors as well — you'd
    duplicate every series under a second set of names. **Go find the real metric
    names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
    they will not match the article's.
  - The trade-off worth being able to argue: collector-side generation is cheaper
    and works before storage, but sees only what passes through that collector;
    Tempo-side generation sees everything that landed, survives collector restarts,
    and can be enabled per-tenant — at the cost of load on the trace store.
 - **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
  click a spike on a latency graph and land in the exact trace. On your platform
  this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
  and the Prometheus datasource declares `exemplarTraceIdDestinations` with
  `name: trace_id`. If you emit your own histograms with exemplars, the label must
  be `trace_id` to match.
 - **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
  can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
  for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
  the pull model is why Prometheus has service discovery and why `up` exists.
 ### Resources
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
+- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
-  Read the first half. Best cardinality explanation available.
+  Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
+  setup, including the hex formatting for trace/span IDs.
-  not a fact.
+- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
+  `OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
-  `servicegraph` connector config and its RED PromQL queries almost verbatim.
+  fields, if you'd rather not hand-roll a processor.
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
+- Udemy backbone: the Loki + LogQL sections.
 - Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
 ### Build
-1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
+1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
-2. Custom application metrics, deliberately few:
+   processor that pulls the current span context and injects `trace_id` / `span_id`.
-   - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
+2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
-   - `ledgerline.transaction.duration` (Histogram; labels: `kind`)
+   handler: request received (or rely on the span), business decision, error.
-   - `ledgerline.fraud.score` (Histogram; no labels)
+3. Add a redaction processor with a denylist of field names.
-   - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
+4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
-   - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
+   with a `container` label (Alloy's relabel rule sets it from the Docker container
-3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
+   name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
-   metrics, so pick a path and implement it:
+5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
-   - *Platform grain:* expose `/metrics` on each service and add them to
+   send some records, and watch nothing appear in Loki. Then check the Alloy UI:
-     `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
+   the receiver shows traffic, and the pipeline shows no logs output. Now extend
-     playbook. Prometheus scrapes your services directly. Note the irony and the
+   `config.alloy.j2` — add a `logs` output on the receiver plus an
-     lesson — you're back to a pull model and a second client library, which is
+   `otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
-     exactly the fragmentation OTel exists to fix.
+   them arrive. Write down what you'd have done to debug this if you *hadn't* known.
-   - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
+   That is the most transferable half-hour in the whole project.
-     Alloy so app metrics flow through the collector as the architecture drawing
+6. Confirm the provisioned correlation works in both directions: Loki's
-     intends. This also closes a real diagram-vs-code gap in the repo.
+   `derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
-   - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
+   (`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
-     `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
+   are verifying, not writing. If a link doesn't appear, the regex didn't match
-     all three signals.
+   your field name — check that before anything else.
-   Do the first to get moving; do one of the others afterwards and write up which
+6. **Migrate the test suite's log-counting helpers.** Three pathology tests
-   you'd choose for a real platform team and why. That write-up is a Phase 9
+   (`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
-   artefact.
+   in `docker compose logs` — the only cross-service call counter that existed at
-4. Deliberately break it: add `account_id` as a label to
+   Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
-   `ledgerline.transactions.total`, run a 200-request burst, and watch the series
+   with LogQL over structured fields instead. This is a small change and a
-   count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
+   pointed one: notice how much shorter and more precise the assertion gets, and
-   you'll never forget.
+   that it now works without shelling out to Docker at all. That delta is the
-5. Compare your hand-written duration histogram against the `spanmetrics`-generated
+   value of structured logging, measured on your own code.
   one. Understand why they differ slightly and when you'd want each.
 6. **Second migration of the call-counting assertions.** The three tests you moved
   to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
   TraceQL query for `sanctions` client spans within a single trace answers
   "how many lookups did this one transaction make?" directly, which is the
   question the test was always really asking. Do it and compare all three
   generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
   clearest possible demonstration of what each signal is actually for, on a
   problem you already understand.
-**Done when:** you have a RED view per service, a working service map, and you can
+**Done when:** from a slow trace you can click into that request's logs, and from
-click a latency spike through to a trace via an exemplar.
+any log line you can click into its trace — in both directions, in two clicks.
@@ -1,76 +1,126 @@
-# Phase 5 — Dashboards and alerts as code
+# Phase 5 — Metrics
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
 **Status:** Not started
-**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
+**Message:** Not blocked technically (metrics don't strictly need tracing
-dashboarding.
+first), but sequenced after Phase 4 per the roadmap's ordering.
-**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
+**Time-box:** 2 evenings.
 recreate in seconds.*
 ### Concepts
- Three levels of "as code", and you should know when each applies:
+- **Instrument types.**
-  1. **File provisioning** — YAML in `provisioning/datasources/` and
+  - **Counter** — monotonic, only increases. Requests, errors, messages consumed.
-     `provisioning/dashboards/`, plus dashboard JSON files mounted into the
+  - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
-     container. Grafana loads them at startup. Simplest, no state, no API tokens,
+    connection-pool checkouts.
-     perfect for a Compose demo. **This alone satisfies the requirement.**
+  - **Histogram** — a distribution. Request latency, payload size. This is what
-  2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
+    gives you percentiles.
-     PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
+  - Synchronous instruments are recorded inline; **observable/asynchronous** ones are
-     function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
+    read via a callback on collection (right for things you *poll*, like pool size).
-     diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
+  - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
-     of Grafonnet (which is legacy and unofficial).
+    **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
-  3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
+    nonsense. Use histograms.
-     multi-environment, or GitOps setups. Overkill here; the right answer *later*,
+- **Cardinality — the one thing that will bite you.** Every unique combination of
-     when this lands on your Kubernetes cluster.
+  metric name + label values is a separate time series in memory. Add `account_id`
- **Dashboards should answer questions, not display data.** A good service dashboard
+  as a label on a 800-account demo and you've made 800 series; do it in production
-  answers, in order: is it up, is it serving, is it erroring, is it slow, and where
+  and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
-  is the pressure. Panels that nobody looks at during an incident are cost, not value.
+  ~10 000 distinct values in production, it does not belong on a metric.** Put it on
- **Alerts as code too.** Grafana alert rules, contact points and notification
+  a span. Histograms multiply this (one series per bucket per label combination), so
-  policies all provision from YAML. And keep the *alert definition* next to the
+  don't over-bucket either.
-  dashboard that explains it.
+  - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
    the concrete path), `http.status_code` class, `kind`, `currency`.
  - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
    raw URL, timestamp, error message.
 - **RED and USE.**
  - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
    per-service dashboard.
  - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
    connection pools, queues, disk. Saturation is the one everyone forgets and the
    one that predicts collapse.
  - You want both: RED tells you the service is slow, USE tells you which resource
    is why.
 - **Span metrics — generate metrics from traces.** Two architectures exist for this,
  and knowing both is the point:
  - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
    `servicegraph` connectors read spans in flight and emit RED metrics and edge
    metrics. This is what most blog posts (including the OneUptime article) show.
  - *In the trace store:* Tempo's **metrics-generator** does the same job after
    ingestion, remote-writing the result to Prometheus. **This is what your platform
    does** — `processors: [service-graphs, span-metrics, local-blocks]` with
    `send_exemplars: true`, remote-writing to a Prometheus started with
    `--web.enable-remote-write-receiver`.
  - So RED metrics, the service map and exemplars **already exist** before you write
    a single metric line. Do not add the collector connectors as well — you'd
    duplicate every series under a second set of names. **Go find the real metric
    names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
    they will not match the article's.
  - The trade-off worth being able to argue: collector-side generation is cheaper
    and works before storage, but sees only what passes through that collector;
    Tempo-side generation sees everything that landed, survives collector restarts,
    and can be enabled per-tenant — at the cost of load on the trace store.
 - **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
  click a spike on a latency graph and land in the exact trace. On your platform
  this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
  and the Prometheus datasource declares `exemplarTraceIdDestinations` with
  `name: trace_id`. If you emit your own histograms with exemplars, the label must
  be `trace_id` to match.
 - **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
  can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
  for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
  the pull model is why Prometheus has service discovery and why `up` exists.
 ### Resources
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
+- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
-  SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
+  Read the first half. Best cardinality explanation available.
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
+- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
-  example.
+  not a fact.
- Udemy backbone: dashboard design sections (panel types, transformations,
+- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
-  variables, dashboard best practice).
+  `servicegraph` connector config and its RED PromQL queries almost verbatim.
 - OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
 ### Build
-Your platform provisions **datasources only** — there is no dashboards provider at
+1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
-all. So this phase is entirely yours, and the right home for it is inside the
+2. Custom application metrics, deliberately few:
-Ansible role, using the same mechanism as everything else.
+   - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
   - `ledgerline.transaction.duration` (Histogram; labels: `kind`)
   - `ledgerline.fraud.score` (Histogram; no labels)
   - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
   - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
 3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
   metrics, so pick a path and implement it:
   - *Platform grain:* expose `/metrics` on each service and add them to
     `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
     playbook. Prometheus scrapes your services directly. Note the irony and the
     lesson — you're back to a pull model and a second client library, which is
     exactly the fragmentation OTel exists to fix.
   - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
     Alloy so app metrics flow through the collector as the architecture drawing
     intends. This also closes a real diagram-vs-code gap in the repo.
   - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
     `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
     all three signals.
   Do the first to get moving; do one of the others afterwards and write up which
   you'd choose for a real platform team and why. That write-up is a Phase 10
   artefact.
 4. Deliberately break it: add `account_id` as a label to
   `ledgerline.transactions.total`, run a 200-request burst, and watch the series
   count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
   you'll never forget.
 5. Compare your hand-written duration histogram against the `spanmetrics`-generated
   one. Understand why they differ slightly and when you'd want each.
 6. **Second migration of the call-counting assertions.** The three tests you moved
   to LogQL in Phase 4 can now count **client spans** in Tempo instead — a
   TraceQL query for `sanctions` client spans within a single trace answers
   "how many lookups did this one transaction make?" directly, which is the
   question the test was always really asking. Do it and compare all three
   generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
   clearest possible demonstration of what each signal is actually for, on a
   problem you already understand.
-1. **Extend the role.** Add
+**Done when:** you have a RED view per service, a working service map, and you can
-   `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
+click a latency spike through to a trace via an exemplar.
   (a file-based dashboard provider pointing at a mounted directory), a
   `config/grafana/dashboards/` directory for the JSON, and the tasks to template
   and copy them. The Compose template already mounts
   `./config/grafana/provisioning` read-only, so the provider is picked up on
   restart.
 2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
   Foundation SDK (Python)**: one function producing a RED dashboard for a given
   service name, looped over the seven services; a second producing a system
   overview (service graph, stream lag, pool saturation, error-budget burn). Commit
   the generator *and* its output; the Ansible role ships the output.
 3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
   Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
   That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
   than a local Makefile target, because it's the same command that builds the
   platform.
 4. While you're in there, review the provisioned datasources and add what's missing:
   `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
   worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
   understand why `filterByTraceID: true` alone is sufficient here, and when it
   wouldn't be.
 5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
 **Done when:** re-running the playbook restores every dashboard from scratch, and no
 dashboard has ever been saved through the Grafana UI.
@@ -1,94 +1,76 @@
-# Phase 6 — Load testing with k6, and percentiles
+# Phase 6 — Dashboards and alerts as code
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
 **Status:** Not started
-**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
+**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
-benefits from Phase 5's dashboards existing.
+dashboarding.
-**Time-box:** 1–2 evenings.
+**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
 recreate in seconds.*
-### Concepts — P95 and P99 properly
+### Concepts
-You asked about these specifically, and there's more to them than "the 95th
+- Three levels of "as code", and you should know when each applies:
-percentile."
+  1. **File provisioning** — YAML in `provisioning/datasources/` and
-
+     `provisioning/dashboards/`, plus dashboard JSON files mounted into the
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
+     container. Grafana loads them at startup. Simplest, no state, no API tokens,
-  is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
+     perfect for a Compose demo. **This alone satisfies the requirement.**
-  is.
+  2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
- **You cannot average percentiles.** Not across instances, not across time buckets.
+     PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
-  `avg(p99)` is wrong and produces a number that looks plausible. The correct move is
+     function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
-  to aggregate the *histogram buckets* and compute the quantile from the sum:
+     diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
-  `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
+     of Grafonnet (which is legacy and unofficial).
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
+  3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
-  your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
+     multi-environment, or GitOps setups. Overkill here; the right answer *later*,
-  bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
+     when this lands on your Kubernetes cluster.
-  bracket your actual SLO threshold. (Native histograms largely solve this and are
+- **Dashboards should answer questions, not display data.** A good service dashboard
-  worth knowing about.)
+  answers, in order: is it up, is it serving, is it erroring, is it slow, and where
- **Which population?** An overall p99 across all endpoints is dominated by whichever
+  is the pressure. Panels that nobody looks at during an incident are cost, not value.
-  endpoint has the most traffic. Slice by route, or you'll miss that
+- **Alerts as code too.** Grafana alert rules, contact points and notification
-  `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
+  policies all provision from YAML. And keep the *alert definition* next to the
- **Tail latency amplification** — the most important idea here for a microservice
+  dashboard that explains it.
  system. If one user request fans out to 10 backend calls, each with an independent
  p99 of 1 s, then the chance the *user request* touches at least one slow call is
  `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
  p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
  fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
  measurable — go measure it.
 - **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
  most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
  that.
 - **Coordinated omission** — a load-generator trap. If your generator waits for a
  response before sending the next request, then when the system stalls, the
  generator stalls too and simply *doesn't record* the requests that would have been
  slow. Your measured latency is systematically optimistic. Fix: use an **open
  model** — a fixed *arrival rate* independent of response times. In k6 that means
  the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
  `constant-vus`. Knowing this is a genuine senior-level differentiator.
 ### Concepts — k6
 - **Test types:** smoke (does it work at all), average-load (normal traffic),
  stress (past normal, find the knee), spike (sudden surge, test recovery),
  soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
  Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
 - **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
  (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
 - **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
  fail the run; **thresholds are SLOs expressed in code and they set the process exit
  code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
 - **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
  view and the application's view sit on the same Grafana time axis. That side-by-side
  is the single most convincing screenshot you can show Babak.
 - k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
  writing 60 lines of it.
 ### Resources
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
+- Grafana docs → *Observability as code* (the index page — it lays out Foundation
-  *Thresholds*. Free, genuinely well written; ~2 h to work through.
+  SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
+- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
-  hands-on).
+  example.
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
+- Udemy backbone: dashboard design sections (panel types, transformations,
- Grafana docs → *k6 output to Prometheus remote write*.
+  variables, dashboard best practice).
 ### Build
-1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
+Your platform provisions **datasources only** — there is no dashboards provider at
-   sharing a `common.js` for setup and payload generation against random accounts.
+all. So this phase is entirely yours, and the right home for it is inside the
-2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
+Ansible role, using the same mechanism as everything else.
 3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
   `http_req_failed rate<0.01`.
 4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
 5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
   sets. This before/after pair is your baseline for Phase 8.
 6. Optional but impressive: have k6 send W3C trace context headers so load-test
   requests appear as traces, letting you click from a k6 spike into a specific
   trace. Check the current k6 docs for the tracing module's status — it has moved
   between experimental and stable.
-**Done when:** `make load-stress` produces a Grafana view where you can see the
+1. **Extend the role.** Add
-arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
+   `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
-service that broke first.
+   (a file-based dashboard provider pointing at a mounted directory), a
   `config/grafana/dashboards/` directory for the JSON, and the tasks to template
   and copy them. The Compose template already mounts
   `./config/grafana/provisioning` read-only, so the provider is picked up on
   restart.
 2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
   Foundation SDK (Python)**: one function producing a RED dashboard for a given
   service name, looped over the seven services; a second producing a system
   overview (service graph, stream lag, pool saturation, error-budget burn). Commit
   the generator *and* its output; the Ansible role ships the output.
 3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
   Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
   That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
   than a local Makefile target, because it's the same command that builds the
   platform.
 4. While you're in there, review the provisioned datasources and add what's missing:
   `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
   worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
   understand why `filterByTraceID: true` alone is sufficient here, and when it
   wouldn't be.
 5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.
 **Done when:** re-running the playbook restores every dashboard from scratch, and no
 dashboard has ever been saved through the Grafana UI.
@@ -1,91 +1,94 @@
-# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
+# Phase 7 — Load testing with k6, and percentiles
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
 **Status:** Not started
-**Message:** Depends on Phase 6's load-test baseline (the latency SLO should
+**Message:** Depends on Phase 5 (metrics to plot k6's output against) and
-come from measured numbers, not be picked from thin air).
+benefits from Phase 6's dashboards existing.
-**Time-box:** 1–2 evenings. Mostly thinking, a little config.
+**Time-box:** 1–2 evenings.
-### Concepts
+### Concepts — P95 and P99 properly
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
+You asked about these specifically, and there's more to them than "the 95th
-  good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
+percentile."
-  `POST /api/transactions` requests that returned 2xx within 500 ms.*
+
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
+- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
-  requests succeed within 500 ms, measured over a rolling 28 days."
+  is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
+  is.
-  Always **looser** than your SLO, so you have room to miss internally without
+- **You cannot average percentiles.** Not across instances, not across time buckets.
-  breaching commercially. If your SLA and SLO are the same number, you've made a
+  `avg(p99)` is wrong and produces a number that looks plausible. The correct move is
-  mistake.
+  to aggregate the *histogram buckets* and compute the quantile from the sum:
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
+  `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
-  budget; over 30 days that's about **43 minutes**. The budget is the thing that
+- **Prometheus classic histograms give you an interpolated estimate**, bounded by
-  turns reliability from an argument into arithmetic: if there's budget left, ship
+  your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
-  features and take risks; if it's spent, the policy says you stop feature work and
+  bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
-  spend on reliability. That policy — agreed in advance by dev, ops and product — is
+  bracket your actual SLO threshold. (Native histograms largely solve this and are
-  the whole point.
+  worth knowing about.)
- **Burn rate.** How fast you're consuming budget relative to the window.
+- **Which population?** An overall p99 across all endpoints is dominated by whichever
-  Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
+  endpoint has the most traffic. Slice by route, or you'll miss that
-  multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
+  `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
-  is what stops you from either paging on every blip or noticing the slow bleed a
+- **Tail latency amplification** — the most important idea here for a microservice
-  month late.
+  system. If one user request fans out to 10 backend calls, each with an independent
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
+  p99 of 1 s, then the chance the *user request* touches at least one slow call is
-  and it's worth being precise because interviewers probe exactly here.
+  `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
-  - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
+  p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
-    from failure to restored service. It drives your architecture: replication,
+  fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
-    warm standby, failover automation, runbook quality.
+  measurable — go measure it.
-  - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
+- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
-    in time: how far back can you afford to lose. It drives your **backup and
+  most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
-    replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
+  that.
-    every 15 minutes.
+- **Coordinated omission** — a load-generator trap. If your generator waits for a
-  - The relationship worth stating out loud: **SLO governs steady-state quality; RTO
+  response before sending the next request, then when the system stalls, the
-    and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
+  generator stalls too and simply *doesn't record* the requests that would have been
-    "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
+  slow. Your measured latency is systematically optimistic. Fix: use an **open
-    5 minutes of transactions." Both feed the SLA.
+  model** — a fixed *arrival rate* independent of response times. In k6 that means
-  - Both are **business decisions, not technical ones** — you derive them by asking
+  the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
-    what an hour of downtime or an hour of lost data actually costs, then buying the
+  `constant-vus`. Knowing this is a genuine senior-level differentiator.
-    cheapest architecture that meets it. Tightening either is expensive and
+
-    non-linear.
+### Concepts — k6
 - **Test types:** smoke (does it work at all), average-load (normal traffic),
  stress (past normal, find the knee), spike (sudden surge, test recovery),
  soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
  Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
 - **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
  (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
 - **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
  fail the run; **thresholds are SLOs expressed in code and they set the process exit
  code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
 - **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
  view and the application's view sit on the same Grafana time axis. That side-by-side
  is the single most convincing screenshot you can show Babak.
 - k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
  writing 60 lines of it.
 ### Resources
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
+- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
-  immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
+  *Thresholds*. Free, genuinely well written; ~2 h to work through.
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
+- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
-  all free PDFs. Run the exercises solo with Ledgerline as the subject.
+  hands-on).
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
+- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
-  free, self-paced, deeper.
+- Grafana docs → *k6 output to Prometheus remote write*.
 - For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
  overview; pair it with your Postgres backup knowledge.
 ### Build
-1. Write `docs/SLO.md`. For Ledgerline define, with justification:
+1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
-   - Availability SLI/SLO on `POST /api/transactions`
+   sharing a `common.js` for setup and payload generation against random accounts.
-   - Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
+2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
-     from thin air)
+3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
-   - A freshness/lag SLI on the notifier consumer group
+   `http_req_failed rate<0.01`.
-   - A stated SLA that is deliberately looser than each SLO
+4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
-   - An RTO and RPO for the Postgres instance, with the backup cadence they imply
+5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
-   - An error budget policy: what happens at 50%, 75% and 100% consumed
+   sets. This before/after pair is your baseline for Phase 9.
-2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
+6. Optional but impressive: have k6 send W3C trace context headers so load-test
-   Your platform ships **Alertmanager** plus a Prometheus rules directory and the
+   requests appear as traces, letting you click from a k6 spike into a specific
-   sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
+   trace. Check the current k6 docs for the tracing module's status — it has moved
-   *Prometheus rule files → Alertmanager*, templated by the Ansible role, with
+   between experimental and stable.
   `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
   be able to argue the alternative: Grafana unified alerting can evaluate across
   datasources (including Loki queries) and lives with your dashboards, whereas
   Alertmanager is metrics-only but is the ecosystem standard and gives you
   grouping, deduplication, inhibition and silences for free. Note which you chose
   and why in `docs/SLO.md`.
 3. Add an error-budget panel to the overview dashboard: budget remaining, current
   burn rate, projected exhaustion.
 4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
   watch the fast-burn alert fire.
-**Done when:** you can explain, in one paragraph each and without notes, the
+**Done when:** `make load-stress` produces a Grafana view where you can see the
-difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
+arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
-decision.
+service that broke first.
@@ -1,90 +1,91 @@
-# Phase 8 — Bottleneck hunting (the capstone)
+# Phase 8 — SLI, SLO, SLA, error budgets, RTO, RPO
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
 **Status:** Not started
-**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
+**Message:** Depends on Phase 7's load-test baseline (the latency SLO should
-dashboards, load-test baselines, and burn-rate alerts are the toolkit this
+come from measured numbers, not be picked from thin air).
 phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
-**Time-box:** 2 evenings. This is where the whole thing pays off.
+**Time-box:** 1–2 evenings. Mostly thinking, a little config.
-### Concepts — how to actually answer "how do I find bottlenecks?"
+### Concepts
-Babak asked how you'd *think* about this, not for a list of tools. Here's the
+- **SLI** — a *measurement* of something users experience, expressed as a ratio of
-method, roughly in order. The discipline is to go **top-down from the user, not
+  good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
-bottom-up from the infrastructure** — bottom-up is how people spend three hours
+  `POST /api/transactions` requests that returned 2xx within 500 ms.*
-tuning a database that wasn't the problem.
+- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
-
+  requests succeed within 500 ms, measured over a rolling 28 days."
-1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
+- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
-   window, compared to what baseline? "It's slow" is not a starting point. "p95 of
+  Always **looser** than your SLO, so you have room to miss internally without
-   `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
+  breaching commercially. If your SLA and SLO are the same number, you've made a
-2. **Localise to a service** using RED + the service graph. Which node's duration
+  mistake.
-   grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
+- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
-   the graph from the leaves inward.
+  budget; over 30 days that's about **43 minutes**. The budget is the thing that
-3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
+  turns reliability from an argument into arithmetic: if there's budget left, ship
-   TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
+  features and take risks; if it's spent, the policy says you stop feature work and
-   several slow traces plus one fast trace of the same endpoint as a control.
+  spend on reliability. That policy — agreed in advance by dev, ops and product — is
-4. **Read the trace for five specific shapes:**
+  the whole point.
-   - **A long span** — the work itself is slow. Look at its attributes: which query,
+- **Burn rate.** How fast you're consuming budget relative to the window.
-     which host, which parameters.
+  Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
-   - **A gap** — wall-clock time not covered by any span. This is the interesting
+  multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
-     one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
+  is what stops you from either paging on every blip or noticing the slow bleed a
-     blocked event loop, or a missing instrumentation library. Ledgerline's
+  month late.
-     `pool_starvation` and `blocking_event_loop` both show up as gaps.
+- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
-   - **Serialisation** — spans laid end to end that could have been concurrent.
+  and it's worth being precise because interviewers probe exactly here.
-     Ledgerline's `no_batching` looks exactly like this.
+  - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
-   - **Repetition** — the same span name 150 times in a row. That's your N+1.
+    from failure to restored service. It drives your architecture: replication,
-   - **Retries** — the same client span repeated 3–4 times with errors. That's
+    warm standby, failover automation, runbook quality.
-     `retry_storm`, and note that it makes an *overload* worse, never better.
+  - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
-5. **Separate latency from saturation.** Switch from RED to USE on the suspected
+    in time: how far back can you afford to lose. It drives your **backup and
-   resource: connection pool in-use vs pool size, stream consumer lag, CPU,
+    replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
-   memory. A resource at 100% utilisation with a growing queue is saturated;
+    every 15 minutes.
-   latency there is a *symptom*, not the cause.
+  - The relationship worth stating out loud: **SLO governs steady-state quality; RTO
-6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
+    and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
-   equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
+    "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
-   concurrent slots. If your pool is 5, you queue — and queueing latency grows
+    5 minutes of transactions." Both feed the SLA.
-   non-linearly as you approach saturation. This one formula explains most
+  - Both are **business decisions, not technical ones** — you derive them by asking
-   connection-pool and worker-count incidents you'll ever see.
+    what an hour of downtime or an hour of lost data actually costs, then buying the
-7. **Check for tail amplification** (Phase 6) before blaming any single service.
+    cheapest architecture that meets it. Tightening either is expensive and
-8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
+    non-linear.
   chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
   over the *same duration*. One variable at a time. Never declare a fix from a
   dashboard that "looks better."
 9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
   becomes the bottleneck. Re-measure after every change; stop when you're inside
   the SLO, not when you run out of ideas.
 10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
    or lock waits inside Postgres, or which line of Python burned the cycles. That's
    where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
    profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
 ### Resources
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
+- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
-  for saturation.
+  immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
+- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
+  all free PDFs. Run the exercises solo with Ledgerline as the subject.
-  thing ever written on why microservice latency behaves the way it does.
+- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
+  free, self-paced, deeper.
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
+- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
-  these; connect them to the trace's DB spans.
+  overview; pair it with your Postgres backup knowledge.
 ### Build
-1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
+1. Write `docs/SLO.md`. For Ledgerline define, with justification:
-   `docs/PATHOLOGIES.md`.**
+   - Availability SLI/SLO on `POST /api/transactions`
-2. Run `make load-average` for 10 minutes.
+   - Latency SLI/SLO on the same (pick the threshold from your Phase 7 baseline, not
-3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
+     from thin air)
-   like a real incident doc: what you observed, what you hypothesised, what you
+   - A freshness/lag SLI on the notifier consumer group
-   checked, what it turned out to be. Include screenshots.
+   - A stated SLA that is deliberately looser than each SLO
-4. When you think you've found them all, run `make chaos-clean`, enable them one at
+   - An RTO and RPO for the Postgres instance, with the backup cadence they imply
-   a time, and confirm each signature.
+   - An error budget policy: what happens at 50%, 75% and 100% consumed
-5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
+2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
-6. Repeat once with a new random set. The second run is where the method actually
+   Your platform ships **Alertmanager** plus a Prometheus rules directory and the
-   becomes yours.
+   sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
   *Prometheus rule files → Alertmanager*, templated by the Ansible role, with
   `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
   be able to argue the alternative: Grafana unified alerting can evaluate across
   datasources (including Loki queries) and lives with your dashboards, whereas
   Alertmanager is metrics-only but is the ecosystem standard and gives you
   grouping, deduplication, inhibition and silences for free. Note which you chose
   and why in `docs/SLO.md`.
 3. Add an error-budget panel to the overview dashboard: budget remaining, current
   burn rate, projected exhaustion.
 4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
   watch the fast-burn alert fire.
-**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
+**Done when:** you can explain, in one paragraph each and without notes, the
-and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
+difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
 decision.
@@ -1,161 +1,90 @@
-# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
+# Phase 9 — Bottleneck hunting (the capstone)
 *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
 **Status:** Not started
-**Message:** Not technically blocked by earlier phases (it stands on its own
+**Message:** Depends on Phases 3–8 all being in place (traces, logs, metrics,
-as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
+dashboards, load-test baselines, and burn-rate alerts are the toolkit this
-re-diagnose in Kibana already exists.
+phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
-**Time-box:** 1–2 evenings.
+**Time-box:** 2 evenings. This is where the whole thing pays off.
-Babak's remaining questions are all *judgment* questions. He isn't testing whether
+### Concepts — how to actually answer "how do I find bottlenecks?"
 you can recite what Filebeat does; he's testing whether you have a framework for
 choosing. Here's the framework, then the answers as worked examples.
-### The framework — six axes to reason on
+Babak asked how you'd *think* about this, not for a list of tools. Here's the
 method, roughly in order. The discipline is to go **top-down from the user, not
 bottom-up from the infrastructure** — bottom-up is how people spend three hours
 tuning a database that wasn't the problem.
-Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
+1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
-
+   window, compared to what baseline? "It's slow" is not a starting point. "p95 of
-1. **Where does the data originate?** Inside your application (you can instrument it)
+   `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
-   or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
+2. **Localise to a service** using RED + the service graph. Which node's duration
-   touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
+   grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
-2. **What shape is it already in?** Structured JSON at the source, or unstructured
+   the graph from the leaves inward.
-   text needing grok/regex parsing? Parsing cost has to land *somewhere*.
+3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
-3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
+   TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
-4. **What's the destination, and how many?** One backend forever, or several, or one
+   several slow traces plus one fast trace of the same endpoint as a control.
-   you expect to change?
+4. **Read the trace for five specific shapes:**
-5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
+   - **A long span** — the work itself is slow. Look at its attributes: which query,
-   on every host) distributes cost and adds resilience but multiplies config surface.
+     which host, which parameters.
-   Parsing centrally concentrates cost and creates a single point of failure — but
+   - **A gap** — wall-clock time not covered by any span. This is the interesting
-   one place to change rules.
+     one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
-6. **What already exists, and what does it cost?** Existing Elastic investment,
+     blocked event loop, or a missing instrumentation library. Ledgerline's
-   existing licences, team skills, and licensing constraints (Elastic's Filebeat and
+     `pool_starvation` and `blocking_event_loop` both show up as gaps.
-   Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
+   - **Serialisation** — spans laid end to end that could have been concurrent.
-   Bit and Vector are permissively licensed). Migration is never free.
+     Ledgerline's `no_batching` looks exactly like this.
-
+   - **Repetition** — the same span name 150 times in a row. That's your N+1.
-### The answers
+   - **Retries** — the same client span repeated 3–4 times with errors. That's
-
+     `retry_storm`, and note that it makes an *overload* worse, never better.
-**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
+5. **Separate latency from saturation.** Switch from RED to USE on the suspected
-from a host or container and forwards them with minimal transformation — and your
+   resource: connection pool in-use vs pool size, stream consumer lag, CPU,
-destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
+   memory. A resource at 100% utilisation with a growing queue is saturated;
-delivery with a registry of file offsets, backpressure handling, and prebuilt modules
+   latency there is a *symptom*, not the cause.
-for common services (nginx, MySQL, system logs) that come with parsing and dashboards
+6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
-included. Reach for it when the source is a file you don't control the format of, the
+   equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
-app can't be modified, and Elasticsearch is where it's going.
+   concurrent slots. If your pool is 5, you queue — and queueing latency grows
-
+   non-linearly as you approach saturation. This one formula explains most
-**When to use Logstash.** When you need real *transformation*: parsing unstructured
+   connection-pool and worker-count incidents you'll ever see.
-text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
+7. **Check for tail amplification** (Phase 7) before blaming any single service.
-splitting or merging events, conditional routing to multiple destinations, or
+8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
-buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
+   chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
-for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
+   over the *same duration*. One variable at a time. Never declare a fix from a
-per host.
+   dashboard that "looks better."
-
+9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
-**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
+   becomes the bottleneck. Re-measure after every change; stop when you're inside
-architecture and the reason is a **division of labour between edge and centre**:
+   the SLO, not when you run out of ideas.
-
+10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
+    or lock waits inside Postgres, or which line of Python burned the cycles. That's
-  backpressure, ships onward. You want the thing running in a thousand places to be
+    where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
-  small and boring.
+    profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
 - Logstash runs on a *few* central nodes: does the expensive grok parsing,
  enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
  you want them in a handful of places you can redeploy, not on a thousand hosts.
 - It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
  persistent queue absorbs it and Filebeat backs off gracefully, rather than every
  host queuing independently or dropping.
 So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
 Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
 need central buffering. Skip Logstash when your logs are already structured JSON and
 Elasticsearch ingest pipelines can do the little parsing that remains.
 **When to use OpenTelemetry.**
 - You own the application code and want telemetry generated *from inside it* — spans
  with business context, custom metrics, logs already carrying trace IDs.
 - You need **more than one signal**, correlated. This is the strongest argument: one
  SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
  threaded through all three. Neither Filebeat nor Logstash does traces.
 - You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
  data model across all of them.
 - You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
  Jaeger by editing Collector YAML.
 - You need distributed tracing at all. There is no serious vendor-neutral alternative.
 - Your telemetry needs processing in flight — redaction, tail sampling, attribute
  rewriting — without touching the app.
 **When *not* to use OpenTelemetry.** This is the question that separates people who
 have used it from people who have read about it.
 - **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
  an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
  file-based collector.
 - **You only need one signal from one source with no correlation.** A single nginx
  access log going to a single log store: OTel is more machinery than the job needs.
  Fluent Bit or Filebeat is simpler and finishes the job today.
 - **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
  Elastic estate with hundreds of Beats modules, dashboards and detection rules is
  years of investment. "We should use OTel because it's the standard" is not a
  business case. Note the direction of travel though: the Collector can already
  receive from and export to most of these, so a *gradual* migration is usually the
  right shape — often by putting a Collector or Vector in front of the existing
  agents rather than ripping them out.
 - **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
  where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
 - **The signal is genuinely business analytics, not telemetry.** Clickstream data for
  a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
  retention, different query patterns, different cost model.
 - **Maturity gaps for a specific need in a specific language.** OTel logs support has
  matured at different rates per language, and profiling is still alpha. Check the
  status page for your language before betting on a signal.
 - **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
  on it. A two-person team that needs error tracking by Friday should install Sentry.
 The senior framing to give Babak: *these tools aren't competitors so much as
 different layers.* OpenTelemetry is a **generation and transport standard** for
 telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
 don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
 Elasticsearch are **stores**. Most real architectures use one from each layer, and
 the interesting question is always which layer a given problem lives in.
 ### Resources
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
+- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
+  for saturation.
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
+- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
-  read these to see how the Collector does the Filebeat/Logstash job itself.
+- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
+  thing ever written on why microservice latency behaves the way it does.
-  caveats.
+- Grafana docs → Tempo → **TraceQL**. You'll live in this.
 - Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
  these; connect them to the trace's DB spans.
-### Build — and this is no longer hypothetical
+### Build
-Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
+1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
-deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
+   `docs/PATHOLOGIES.md`.**
-Filebeat reading Docker container logs and Logstash parsing JSON into daily
+2. Run `make load-average` for 10 minutes.
-`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
+3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
-questions — the codebase he handed you contains both worlds.
+   like a real incident doc: what you observed, what you hypothesised, what you
   checked, what it turned out to be. Include screenshots.
 4. When you think you've found them all, run `make chaos-clean`, enable them one at
   a time, and confirm each signature.
 5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
 6. Repeat once with a new random set. The second run is where the method actually
   becomes yours.
-1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
+**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
-   flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
+and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
   Alloy → Loki. Compare them on the same data.
 2. Test the direct ingest path the README documents — pipe a JSON line into
   Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
   parsing happens before you change it.
 3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
   that parses them. Then do the same job with Alloy's `loki.process` stages. Time
   both. Note which one you'd rather debug at 2 a.m.
 4. Compare the *investigation experience* directly: take one incident you already
   diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
   you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
   wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
   for some questions, and Loki's cost model genuinely beats Elasticsearch's for
   others.
 5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
   licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
   permissively licensed. For some employers that alone decides it.
 6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
   valuably — what you actually found annoying about each. Having run both stacks
   against the same application is the difference between reciting a comparison and
   arguing one.
@@ -6,7 +6,7 @@ for the phase you're actually working on. This file exists so an agent (or
 you) can check overall status and sequencing without loading every phase's
 full detail.*
-Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase
+Twelve phases (0 through 11 — Phase 0 has two halves, 0a and 0b). Each phase
 file follows the same shape: **Status**, **Message** (what's done, notes,
 warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
@@ -16,56 +16,60 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
 |---|---|---|---|---|---|
 | 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
 | 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
-| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
+| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
-| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
+| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
-| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
+| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
-| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
+| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
-| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
+| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
-| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
+| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
-| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
+| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
-| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
+| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
-| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
+| 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
-| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
+| 10 | [`phase-10.md`](phase-10.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
 | 11 | [`phase-11.md`](phase-11.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
 ## What each phase is, in one line
 - **0 — Build the base project, test-first.** Write the black-box test suite,
  then implement the seven-service system against it as a frozen contract.
  **Done** — see `phase-0.md` for the full implementation report.
- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
+- **1 — Deployment.** Put Ledgerline on a server with Docker Compose behind
  Traefik and a Let's Encrypt certificate, served at a real domain. Plumbing
  only — no telemetry, no app changes. **Done** — see `phase-1.md`.
 - **2 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
  split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
  Ansible and get one auto-instrumented trace into Tempo.
- **2 — Distributed tracing.** The heart of the task: auto- and
+- **3 — Distributed tracing.** The heart of the task: auto- and
  manual-instrument all seven services, and hand-propagate trace context
  across the Redis Stream hop, which carries none by default.
- **3 — Structured logs, correlated to traces.** Replace prose logging with
+- **4 — Structured logs, correlated to traces.** Replace prose logging with
  structured events carrying a lowercase `trace_id`, so Loki and Tempo link
  in both directions.
- **4 — Metrics.** Instrument RED metrics per service, understand cardinality
+- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
  limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
  silently. Tempo's metrics-generator already produces span metrics and the
  service graph — don't duplicate it in the collector.
- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the
+- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
  Grafana Foundation SDK and ship it through the Ansible role, so "destroy
  and recreate in seconds" is a real, repeatable command.
- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't
+- **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
  be averaged, what coordinated omission is, and drive Ledgerline with
  arrival-rate executors to get a clean before/after load profile.
- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
+- **8 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
-  SLOs from the Phase 6 baseline (not from thin air), write an error-budget
+  SLOs from the Phase 7 baseline (not from thin air), write an error-budget
  policy, and wire multi-burn-rate alerts.
- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
+- **9 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
  of chaos flags, find the pathologies from telemetry alone using a top-down,
  hypothesis-driven method, and write it up as an incident doc.
- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
+- **10 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
  OpenTelemetry on six concrete axes, using the platform's real second stack
  (Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
- **10 — Package the delivery.** README, a rehearsed five-minute demo, the
+- **11 — Package the delivery.** README, a rehearsed five-minute demo, the
  written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`,
  `CONCEPTS.md`), and an honest "what I'd do differently in production"
  section.
 ## If you have to cut scope
-Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not
+Cut Phase 10's build (keep the writing) and Phase 7's soak test first. **Do not
-cut Phase 8** — it's the only phase that proves you learned anything.
+cut Phase 9** — it's the only phase that proves you learned anything.