diff --git a/CLAUDE.md b/CLAUDE.md index 43cfe43..d03e78a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,9 +16,9 @@ Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is assessed by a mentor, Babak. The full plan lives in `docs/roadmap/`, split one file per phase so an agent -working on a given phase doesn't have to load the other ten: `intro.md` +working on a given phase doesn't have to load the other eleven: `intro.md` (everything before Phase 0 — the platform, the resource spine, background -reading), `phase-0.md` through `phase-10.md`, and `summary.md` (a one-page +reading), `phase-0.md` through `phase-11.md`, and `summary.md` (a one-page status/sequencing overview of all of them). **Start every session with `docs/roadmap/summary.md`** to see where the user is — see "Current phase" below. Phase 0 is built test-first from @@ -34,7 +34,7 @@ It deploys two stacks on a target host via Docker Compose: - **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with **Grafana Alloy** as the collector and node-exporter for host metrics. - **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the - hands-on subject of Phase 9, not part of the main pipeline. + hands-on subject of Phase 10, not part of the main pipeline. Five facts about that platform that change what correct code looks like here. Verify them against the live system rather than assuming — they were read from the configs @@ -91,22 +91,22 @@ These override helpfulness. Breaking any of them ruins the exercise. 1. **Do not read, summarise, quote, paraphrase, or hint at the contents of `docs/PATHOLOGIES.md`** unless the user explicitly says they have finished - Phase 8 and wants to review it. Not even "let me check the docs to confirm." -2. **Do not reveal which chaos flags are currently enabled** during Phase 8. Do + Phase 9 and wants to review it. Not even "let me check the docs to confirm." +2. **Do not reveal which chaos flags are currently enabled** during Phase 9. Do not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and do not infer the set from the source and announce it. If the user asks you to check, remind them what phase they're in and let them insist — then comply. -3. **Do not run `pytest -m pathology` during Phase 8.** The pathology suite has +3. **Do not run `pytest -m pathology` during Phase 9.** The pathology suite has one named test per flag, so its pass/fail output is a complete answer key. Same for `make test-pathology` and `make test-slow`. Running the `contract`, `seed`, `baseline` or `guard` markers is fine. -4. **Do not diagnose the system for the user during Phase 8.** That phase is the +4. **Do not diagnose the system for the user during Phase 9.** That phase is the capstone: they find the bottlenecks from telemetry alone. Your job there is to teach *method* (how to read a trace gap, how to reason about saturation) and to operate tools they ask you to operate — not to name the culprit. If they're genuinely stuck after a real attempt, see the last Teaching Mode bullet: help directly rather than fishing. -5. Outside Phase 8 these relax. Check `docs/roadmap/summary.md` for the current +5. Outside Phase 9 these relax. Check `docs/roadmap/summary.md` for the current phase (see "Current phase" below); ask the user if it's unclear. --- @@ -119,7 +119,7 @@ spoiler rules, Teaching Mode scope, what to build next, or any "where am I / what's next" question), **read `docs/roadmap/summary.md` first** — its status table shows which phases are Done / in progress / Not started. Then read only the `phase-N.md` file for the phase in play (and `intro.md` for platform -background if needed); don't load all eleven. +background if needed); don't load all twelve. - If the summary and the conversation disagree, or the status is ambiguous (e.g. no phase is marked in progress), ask the user before assuming. @@ -145,10 +145,10 @@ look met: lines) in the same change, so the two never disagree. Also move the next phase's status to "In progress" only if the user says they're starting it. -Phase reference: **0a** test suite · **0b** implement to green · 1 concepts + -first trace · 2 tracing · 3 logs · 4 metrics · 5 dashboards-as-code · -6 k6 + percentiles · 7 SLO/error budget · 8 bottleneck hunt · -9 Filebeat/Logstash/OTel judgment · 10 delivery. +Phase reference: **0a** test suite · **0b** implement to green · 1 deployment +(Compose + Traefik) · 2 concepts + first trace · 3 tracing · 4 logs · 5 metrics · +6 dashboards-as-code · 7 k6 + percentiles · 8 SLO/error budget · +9 bottleneck hunt · 10 Filebeat/Logstash/OTel judgment · 11 delivery. --- @@ -218,8 +218,8 @@ Decide by what's being touched: **Phase 0 is explicitly exempt**, in both halves. The base project was delegated to coding agents by design: **0a** writes the tests, **0b** implements against -them. Build both from their specs without teaching. Phases 1–10 are the learning -work. +them. Build both from their specs without teaching. **Phase 1 (deployment)** is +likewise plumbing — just build it. Phases 2–11 are the learning work. If the user says "just do it," "give me the code," or "skip the socratic thing" — comply immediately and without negotiation for that work item. @@ -243,7 +243,7 @@ split collapses if the tests drift toward the implementation. counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception paths) are deliberately built so faking is harder than implementing; treat that as the intent for all of them. -- From Phase 3 onward the log-line-counting assertions in the pathology suite +- From Phase 4 onward the log-line-counting assertions in the pathology suite *should* be rewritten against structured logs and then span/metric data. That is a planned migration the user drives — propose it, don't do it unasked, and it does not license loosening anything else. @@ -287,7 +287,7 @@ docker compose logs -f gateway # actual log shape, not the intended one make test-contract # fast functional suite (< 60s) make test-baseline # is the clean system actually healthy -pytest tests/ -m guard -q # "no observability yet" — delete at Phase 2 +pytest tests/ -m guard -q # "no observability yet" — delete at Phase 3 git diff --stat tests/ # did anything touch the contract? should be empty ``` @@ -379,7 +379,7 @@ Things a well-meaning assistant does wrong here: - **Importing application code into `tests/`.** The suite is black-box on purpose; an import couples it to implementation and the contract stops meaning anything. -- **Writing `common/telemetry.py` for him** in Phase 2. That file is the single +- **Writing `common/telemetry.py` for him** in Phase 3. That file is the single most valuable thing he writes. Checkpoint it: Resource → TracerProvider → BatchSpanProcessor → exporter → propagator, one at a time. - **Creating or editing dashboards through the Grafana UI or HTTP API** as a @@ -400,13 +400,13 @@ Things a well-meaning assistant does wrong here: Aggregate the histogram buckets, then compute the quantile. - **High-cardinality metric labels** — `account_id`, `transaction_id`, `merchant_name`, raw paths. These belong on spans. (There is one *deliberate* - cardinality-explosion exercise in Phase 4; that one is intentional and gets + cardinality-explosion exercise in Phase 5; that one is intentional and gets reverted.) - **`constant-vus` for load profiles** where an open model is correct. Closed-model executors hide latency through coordinated omission. - **Putting `trace_id` in a Loki label** instead of the log body + derived field. - **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a - Phase 9 judgment exercise and a small side pipeline only — not the + Phase 10 judgment exercise and a small side pipeline only — not the architecture. - Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md` for him. Those are his reasoning. Review, challenge, poke holes — don't write. @@ -423,7 +423,7 @@ Things a well-meaning assistant does wrong here: `db.system`, `server.address`). Namespace custom ones as `ledgerline.*`. - `service.name` matches the directory name under `services/`. - Duplication across services is fine. Don't refactor toward a shared framework. -- Docker Compose only, locally, through Phase 9. Kubernetes is a later +- Docker Compose only, locally, through Phase 10. Kubernetes is a later follow-up — porting instrumentation and debugging a cluster at the same time is how this stalls. - Never save a Grafana dashboard from the UI. diff --git a/README.md b/README.md index 60e8d83..06517a2 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ make test-contract # fast functional suite (~60s) Then open **http://localhost:8000**. The observability backend lives in a separate Ansible repo and is deployed to its -own host. From Phase 1 onward: +own host. From Phase 2 onward: ```bash # in observability-ansible/ @@ -88,7 +88,7 @@ Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz` (dependency check). **Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no -automatic trace-context propagation, so Phase 2 has to implement it by hand. +automatic trace-context propagation, so Phase 3 has to implement it by hand. That's the most valuable exercise in the project. Don't swap it out. ### The primary request flow @@ -137,7 +137,7 @@ Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`, `no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`, `chatty_health`. -> **During Phase 8, don't open the chaos panel and don't call +> **During Phase 9, don't open the chaos panel and don't call > `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the > seed; the point is to find the pathologies from telemetry alone. The panel > will happily tell you the answer, which is the one thing you don't want. @@ -160,8 +160,8 @@ Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`, | `test-slow` | The soak-ish pathology tests | | `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile | | `fix-index` / `drop-index` | Add or remove the deliberately missing indexes | -| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 5+* | -| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 6+* | +| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 6+* | +| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 7+* | --- @@ -198,17 +198,17 @@ make test # everything except slow ``` > ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so -> its output is a complete answer key. Don't run it during the Phase 8 +> its output is a complete answer key. Don't run it during the Phase 9 > bottleneck hunt. `tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no `prometheus_client`, no structured logging, no `/metrics`. **Delete it at the -start of Phase 2**; it has done its job by then. +start of Phase 3**; it has done its job by then. Some pathology assertions currently count log lines, because plain text is the only cross-service observability surface that exists at Phase 0. That is the deficiency this project sets out to fix — those assertions get rewritten against -structured logs in Phase 3 and span data in Phase 4, and become both simpler and +structured logs in Phase 4 and span data in Phase 5, and become both simpler and more robust. See `tests/README.md`. --- @@ -321,14 +321,14 @@ Rehearse this before showing anyone. └─ docs/ ├─ roadmap/ # the ten-phase learning plan, one file per phase │ ├─ intro.md # platform background, resource spine (read first) - │ ├─ phase-0.md … phase-10.md # one phase per file — status, message, concepts, build + │ ├─ phase-0.md … phase-11.md # one phase per file — status, message, concepts, build │ └─ summary.md # one-page status/sequencing overview of all phases ├─ phase-0a-test-suite-spec.md # spec the test suite was written from ├─ phase-0-base-project-spec.md # spec the base app was built from ├─ PATHOLOGIES.md # ⚠️ spoilers — the answer key ├─ SLO.md # SLIs, SLOs, SLA, RTO/RPO, error budget policy ├─ CONCEPTS.md # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words - ├─ INVESTIGATION.md # the Phase 8 bottleneck hunt, written as an incident doc + ├─ INVESTIGATION.md # the Phase 9 bottleneck hunt, written as an incident doc └─ PIPELINE-CHOICES.md # Filebeat vs Logstash vs OpenTelemetry — when and why ``` @@ -338,16 +338,17 @@ Rehearse this before showing anyone. - [ ] **Phase 0a** — integration test suite, written first, red against nothing - [ ] **Phase 0b** — base project implemented until the suite is green -- [ ] **Phase 1** — concepts, first trace end to end -- [ ] **Phase 2** — distributed tracing across all seven services -- [ ] **Phase 3** — structured logs, correlated with traces -- [ ] **Phase 4** — metrics, RED, service map, exemplars -- [ ] **Phase 5** — dashboards and alerts as code -- [ ] **Phase 6** — k6 load testing, percentiles -- [ ] **Phase 7** — SLIs, SLOs, error budgets, RTO/RPO -- [ ] **Phase 8** — bottleneck hunt -- [ ] **Phase 9** — Filebeat / Logstash / OpenTelemetry judgment -- [ ] **Phase 10** — delivery +- [x] **Phase 1** — deployment: Docker Compose + Traefik on a server +- [ ] **Phase 2** — concepts, first trace end to end +- [ ] **Phase 3** — distributed tracing across all seven services +- [ ] **Phase 4** — structured logs, correlated with traces +- [ ] **Phase 5** — metrics, RED, service map, exemplars +- [ ] **Phase 6** — dashboards and alerts as code +- [ ] **Phase 7** — k6 load testing, percentiles +- [ ] **Phase 8** — SLIs, SLOs, error budgets, RTO/RPO +- [ ] **Phase 9** — bottleneck hunt +- [ ] **Phase 10** — Filebeat / Logstash / OpenTelemetry judgment +- [ ] **Phase 11** — delivery Full plan: [`docs/roadmap/`](docs/roadmap/) — start with [`intro.md`](docs/roadmap/intro.md), then the phase you're on, or diff --git a/docker-compose.traefik.yml b/docker-compose.traefik.yml new file mode 100644 index 0000000..5b01949 --- /dev/null +++ b/docker-compose.traefik.yml @@ -0,0 +1,63 @@ +# Production-style override: put the gateway (frontend + API) behind Traefik +# with automatic Let's Encrypt TLS. Use together with the base file: +# +# docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --build +# +# Only Traefik publishes host ports (80/443). Everything else is reachable +# only on the compose network. + +services: + traefik: + image: traefik:v3.6 + restart: unless-stopped + command: + - --providers.docker=true + - --providers.docker.exposedByDefault=false + - --entrypoints.web.address=:80 + - --entrypoints.web.http.redirections.entryPoint.to=websecure + - --entrypoints.web.http.redirections.entryPoint.scheme=https + - --entrypoints.websecure.address=:443 + - --certificatesresolvers.le.acme.httpchallenge=true + - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web + - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json + - --log.level=INFO + - --accesslog=true + ports: + - "80:80" + - "443:443" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - letsencrypt:/letsencrypt + + gateway: + ports: !reset [] + labels: + - traefik.enable=true + - traefik.http.routers.ledgerline.rule=Host(`ledgerline.vione.cloud`) + - traefik.http.routers.ledgerline.entrypoints=websecure + - traefik.http.routers.ledgerline.tls.certresolver=le + - traefik.http.services.ledgerline.loadbalancer.server.port=8000 + + postgres: + ports: !reset [] + + redis: + ports: !reset [] + + ledger: + ports: !reset [] + + fraud: + ports: !reset [] + + sanctions: + ports: !reset [] + + insights: + ports: !reset [] + + notifier: + ports: !reset [] + +volumes: + letsencrypt: diff --git a/docs/roadmap/intro.md b/docs/roadmap/intro.md index 1d4dd2c..b945241 100644 --- a/docs/roadmap/intro.md +++ b/docs/roadmap/intro.md @@ -1,7 +1,7 @@ # Observability & APM — Delivery Roadmap for Babak's Task *Part of `docs/roadmap/` — platform background and reading guidance shared by -every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`; +every phase. Per-phase detail lives in `phase-0.md` through `phase-11.md`; [`summary.md`](summary.md) gives the one-page overview and status of all of them.* @@ -90,16 +90,16 @@ learn OTel first, every backend becomes "point OTLP at a different URL." One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly early, in parallel with OTel metrics — because it *constrains what you're allowed to emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them -(Phases 4–5), not up front. +(Phases 5–6), not up front. Filebeat and Logstash are a **different world entirely** (Elastic's). They are not -part of the Grafana stack and you do not need them for this task. Phase 9 covers +part of the Grafana stack and you do not need them for this task. Phase 10 covers them as a judgment question, which is exactly how Babak framed them. ### Verdict on the OneUptime article you found **Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a -teaching resource. Read it in **Phase 4/5**, not now; it assumes you already +teaching resource. Read it in **Phase 5/6**, not now; it assumes you already understand spans, span metrics and PromQL. What's genuinely good in it, and rare to find in one place: @@ -133,7 +133,7 @@ Two caveats: This changes *where* your telemetry goes and *how the pipeline is configured*. It does not change the learning arc, the app, or the test suite. Read this before -Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9. +Phase 2; it rewrites parts of Phases 2, 4, 5, 6, 8 and 10. ### What the repo is @@ -199,7 +199,7 @@ Prometheus`. The shipped config does not do that: Neither is wrong; the diagram shows the intended architecture and the code implements a simpler one. Closing that gap — adding `prometheus.scrape` + `prometheus.remote_write` to Alloy so metrics really do flow through the collector — -is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth +is a genuinely good Phase 5 exercise, and **noticing the discrepancy at all is worth mentioning to Babak.** Spotting that a diagram and its implementation have drifted is exactly the skill this task is meant to build. @@ -264,7 +264,7 @@ Tempo and remote-written into Prometheus (which runs with connectors to Alloy** — you'd be generating the same series twice under different names. The metric names differ from the OneUptime article's; go find the actual names in Prometheus before writing any PromQL. (`local-blocks` also gives you -TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.) +TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.) **3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki datasource is provisioned with: @@ -295,16 +295,17 @@ your deliverable, and the natural home for it is *inside this Ansible role*. | Phase | Impact | |---|---| | 0 / 0a | **None.** App and test suite are unaffected. | -| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. | -| 2 | Endpoint changes to Alloy; tracing work itself is identical. | -| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. | -| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. | -| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. | -| 6 | None. | -| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. | -| 8 | None, plus TraceQL metrics from `local-blocks`. | -| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. | -| 10 | Demo now includes `ansible-playbook` rebuilding the platform. | +| 1 | **New.** Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes. | +| 2 | Replaced. Deploy the real stack with Ansible instead of running a single container. | +| 3 | Endpoint changes to Alloy; tracing work itself is identical. | +| 4 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. | +| 5 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. | +| 6 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. | +| 7 | None. | +| 8 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. | +| 9 | None, plus TraceQL metrics from `local-blocks`. | +| 10 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. | +| 11 | Demo now includes `ansible-playbook` rebuilding the platform. | ### Where to run it @@ -326,7 +327,7 @@ Is this repo a **platform you must consume as-is**, or a **starting point you ma modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and extend `group_vars`. If Babak intends it as a fixed platform, several -recommendations below change — ask before Phase 3. +recommendations below change — ask before Phase 4. --- @@ -340,8 +341,8 @@ subtitle-to-notes workflow. | Resource | Format | Length | Cost | Best for | |---|---|---|---|---| -| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. | -| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. | +| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 4–6. Comes with browser-based Killercoda playgrounds. | +| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 3–5. | If budget allows, buy both — they overlap by maybe 30% and the second half of each is different. If only one: take the **OpenTelemetry** one, because instrumentation @@ -351,27 +352,27 @@ is the part you have to *write*, and Grafana's own docs are excellent for the re | Resource | Format | Length | Use in | |---|---|---|---| -| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. | -| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. | -| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. | +| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 2. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. | +| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 3. The best Python-specific overview, with a companion repo. | +| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 3. Auto- vs manual-instrumentation side by side. | | **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. | -| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. | -| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. | -| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 | -| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. | -| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. | -| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on | -| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 | -| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. | -| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. | -| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 | -| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. | -| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 | -| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 | +| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 2. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. | +| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 2–5. The neutral source of truth. | +| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 6–7 | +| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 2–5. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. | +| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 5, 9. This is where your RED metrics and service graph actually come from. | +| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 10, now hands-on | +| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 8 | +| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 8. Self-run the exercises with Ledgerline as the subject service. | +| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 8, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. | +| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 9 | +| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 5. The clearest explanation anywhere of what cardinality actually costs. | +| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 3 | +| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 10 | -**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on +**Where to run it:** Docker Compose on your laptop for Phases 0–9. Don't put this on your k8s cluster until everything works locally — debugging instrumentation and debugging Kubernetes at the same time is how these projects die. Porting it to the -cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half +cluster is a good Phase 11+ follow-up, and it's the natural bridge to the DevOps half Babak has planned for you. diff --git a/docs/roadmap/phase-0.md b/docs/roadmap/phase-0.md index 8ff9238..69229d0 100644 --- a/docs/roadmap/phase-0.md +++ b/docs/roadmap/phase-0.md @@ -67,7 +67,7 @@ deliberate brokenness. ### Why TDD here, when the app itself doesn't matter -The microservices are throwaway. The *properties* aren't. Everything from Phase 2 +The microservices are throwaway. The *properties* aren't. Everything from Phase 3 onward assumes the system still behaves a specific way: seven real services in the path, async work genuinely off the request path, five hot accounts owning a third of the rows, and thirteen pathologies that actually degrade something. You @@ -114,13 +114,13 @@ Done when `make test` is green three consecutive runs and ### Then, yourself 1. **Read every service's `app.py` end to end.** You need to know this codebase - cold, because from Phase 2 onward you're modifying all seven services. + cold, because from Phase 3 onward you're modifying all seven services. 2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and then try to figure out what's wrong using only `docker compose logs`.** Give it twenty real minutes. You will fail, and the specific way you fail — no timing, no causality, no cross-service correlation, a `X-Request-Id` that dies at the Redis Stream — is the motivation for everything that follows. Write down what - you wished you had. That list is your acceptance criteria for Phase 8. + you wished you had. That list is your acceptance criteria for Phase 9. **Done when:** `make test` is green three runs in a row, all nine acceptance criteria in the base spec pass, and you have that written list of frustrations. diff --git a/docs/roadmap/phase-1.md b/docs/roadmap/phase-1.md index e5ff039..bf4e3a9 100644 --- a/docs/roadmap/phase-1.md +++ b/docs/roadmap/phase-1.md @@ -1,89 +1,74 @@ -# Phase 1 — The model, and one signal end to end +# Phase 1 — Deployment: Ledgerline on a server behind Traefik *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)* -**Status:** Not started +**Status:** Done -**Message:** Blocked on nothing — Phase 0 is done and the platform is -described above. Waiting to begin. +**Message:** Ledgerline is running on a remote server and served over HTTPS at +**https://ledgerline.vione.cloud**. Verified against the live deployment, not +just the config: the frontend loads, `/api/health/all` reports every service +`ok`, and the origin serves a real Let's Encrypt certificate (valid until +2027-01-07, auto-renewed by Traefik). -**Time-box:** one evening. +- **Target:** the throwaway server `czy` (Ubuntu 24.04, 1 vCPU, 1 GB RAM), repo + at `/opt/apps/LedgerLine`. Docker 29.9 and Compose v5.6 installed from + Docker's apt repo. A 2 GB swapfile was added (persisted in `/etc/fstab`) — + nine containers do not fit in 1 GB of RAM without it, and steady state still + uses roughly 700 MB plus swap. +- **How it's wired:** `docker-compose.traefik.yml` is an *override*, not an + edit of `docker-compose.yml`, so local development is unchanged. It adds + Traefik v3.6 on ports 80/443 (Docker provider, `exposedByDefault=false`, + HTTP→HTTPS redirect, Let's Encrypt via HTTP-01), routes + ``Host(`ledgerline.vione.cloud`)`` to the gateway on port 8000, and strips + the published host ports from every other service (`ports: !reset []`) so + Postgres, Redis and the services are reachable only on the Compose network. +- **No application or `tests/` changes.** The Dockerfile and base compose file + are untouched; schema applied and data seeded on the server with the same + steps as `make seed` (800 accounts, 250,000 transactions). +- **Deploy / redeploy command** (always both files — a bare `docker compose up` + would use the base file alone and publish every port): -**Why:** you need the vocabulary, but more importantly you need to see one trace -appear in Grafana within an hour, so the rest of the roadmap feels concrete rather -than abstract. + ```bash + docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --build + ``` + +- **Cloudflare gotchas hit on the way:** + - The first certificate attempt failed with an ACME 404 because the + Cloudflare DNS record did not yet point at the server — a packet capture + on the server showed no inbound traffic from Cloudflare at all. Traefik + only retries issuance on restart, so after fixing the record it needed + `docker compose ... restart traefik`. + - With the orange-cloud proxy on, the SSL/TLS mode must be **Full** — + **Flexible** combined with Traefik's HTTPS redirect causes a redirect loop. +- **Caveats to remember:** the deployment is public, including the + `/api/admin/chaos` endpoints (anyone who finds them can flip flags); + internet scanners probed the site within seconds of it going live. The + server is disposable by design, and Traefik's state lives in the + `letsencrypt` volume, which is lost on an OS reinstall (a new certificate is + simply issued on the next run). + +**Time-box:** 1 evening. + +**Deliverable:** the unmodified-behaviour Ledgerline system reachable at a real +domain over HTTPS, reproducible with one compose command. ### Concepts -- **Monitoring vs observability.** Monitoring answers questions you knew to ask - (known unknowns): dashboards and alerts built from predefined aggregates. - Observability is a property of the system: can you ask a *new* question about - production, that you never anticipated, without shipping new code? Charity Majors' - framing — monitoring is known-unknowns, observability is unknown-unknowns. The - practical implication for you: emit **wide, high-dimensional events** (rich span - attributes), not just pre-aggregated counters. -- **The OTel architecture split: API vs SDK vs Collector.** - - The **API** is what your code and third-party instrumentation libraries call. - If no SDK is configured, API calls are no-ops — that's deliberate, it means a - library can be instrumented without forcing a dependency on any implementation. - - The **SDK** is the engine: sampling, batching, context propagation, export. - Application owners configure the SDK; library authors only touch the API. - - The **Collector** is a standalone process that receives, processes and exports - telemetry. Optional, but you should use one. -- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318. -- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`, - `service.version`, `deployment.environment`) and is attached to every signal from - that process. Attributes describe the individual span/metric/log. -- **Semantic conventions**: standardised attribute names (`http.request.method`, - `db.system`, `server.address`). Use them for anything standard; namespace your own - (`ledgerline.account.id`). This is what makes cross-service, cross-language - correlation work — and it's what Grafana's prebuilt dashboards key off. -- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha — - aware of, not learning yet.) -- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your - app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of - seven services. The Collector also does the things you don't want in your app: - retry, PII redaction, tail sampling, memory limiting, fan-out to multiple - destinations. - -### Resources - -- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the - M.E.L.T. and context-propagation sections; skip the Node code. -- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min. -- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop - repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand - it — just see what "done" looks like. - -### Build - -1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it, - `ansible-galaxy collection install -r requirements.yml`, change - `grafana_admin_password` (vault it), then - `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the - README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093, - Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318. - Then open the Alloy UI and **read the component graph** — it shows you the exact - pipeline, including the fact that the OTLP receiver has only a traces output. - Seeing that in the UI before you get bitten by it is worth ten minutes. - *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`, - a single container with everything pre-wired. Useful for a first trace, but - don't build on it — its pipeline differs from your real platform in exactly the - ways that matter.)* -2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code - auto-instrumentation: - ``` - pip install opentelemetry-distro opentelemetry-exporter-otlp - opentelemetry-bootstrap -a install - ``` - and change its command to - `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003` - with `OTEL_SERVICE_NAME=sanctions`, - `OTEL_EXPORTER_OTLP_ENDPOINT=http://:4318`, - `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at - Tempo directly. -3. Fire a request. Open Grafana → Explore → Tempo → find the trace. - -**Done when:** you can see a single-service trace in Tempo, and you can explain out -loud what the Resource attributes on it mean and where they came from. +- **Traefik's model:** *entrypoints* (the ports it listens on), *routers* + (rule → service), *services* (where to send traffic), *middlewares*, and + *providers* (where config comes from — here, Docker labels on containers). + Compare with nginx, where the same information lives in one static config + file. +- **Compose override files.** `-f base -f override` merges the files; + `!reset` clears an inherited key (here, `ports`) instead of appending to it. +- **ACME HTTP-01 behind a CDN.** Let's Encrypt must be able to reach port 80 on + the *hostname's actual origin*. With Cloudflare proxying, that only works + once the DNS record points at the right server. +- **Why this phase exists before observability.** The telemetry work in + Phases 2–11 is more convincing when the system under observation is a real + deployment rather than a laptop, and the platform (`observability-ansible`) + is already a remote deployment — Ledgerline now sits on the same footing. +**Done when:** `https://ledgerline.vione.cloud` shows the frontend with a valid +certificate; `/api/health/all` is all `ok`; only ports 22, 80 and 443 are +reachable on the server; and the deploy is one documented compose command. diff --git a/docs/roadmap/phase-10.md b/docs/roadmap/phase-10.md index 22a0f23..77060cd 100644 --- a/docs/roadmap/phase-10.md +++ b/docs/roadmap/phase-10.md @@ -1,40 +1,161 @@ -# Phase 10 — Package the delivery +# Phase 10 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry -*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)* +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`phase-11.md`](phase-11.md)* **Status:** Not started -**Message:** Depends on every prior phase being done — this is the wrap-up, -not independent work. +**Message:** Not technically blocked by earlier phases (it stands on its own +as a judgment exercise), but sequenced last-but-one so Phase 9's incident to +re-diagnose in Kibana already exists. -**Time-box:** one evening. +**Time-box:** 1–2 evenings. -Babak gave you a task; deliver it like an engineer, not like a student. +Babak's remaining questions are all *judgment* questions. He isn't testing whether +you can recite what Filebeat does; he's testing whether you have a framework for +choosing. Here's the framework, then the answers as worked examples. -1. **`README.md`** — architecture diagram, one-command start, what each container is - for, and a "here's the demo" script. -2. **The five-minute demo path**, rehearsed: - - Delete every dashboard in the Grafana UI, then `ansible-playbook - playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt - from code, one command. - - Submit one transaction from the frontend, then follow it through Grafana: RED - dashboard → latency spike → exemplar → trace → span → logs for that span → back - to the trace. One request, all three signals, no searching. - - `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire, - and the service map turn red at a specific node. - - Show `docs/INVESTIGATION.md` and how you found the pathologies. -3. **The written answers** Babak asked for, as documents in the repo, not as a - conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`, - `docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99, - RTO/RPO/SLA/SLO/error budget in your own words. -4. **The test suite as part of the story.** Show `make test` green, then show the - three generations of the call-counting assertion side by side — - `docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what - each signal bought you. Most people demo dashboards; almost nobody can show - their observability work paying off inside their own test suite. -5. **A "what I'd do differently in production" section.** Tail sampling instead of - 100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet; - log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what - you *didn't* do, and why, is the strongest signal of seniority in the whole - delivery. +### The framework — six axes to reason on + +Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through: + +1. **Where does the data originate?** Inside your application (you can instrument it) + or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't + touch, syslog from a switch)? You can only use OTel SDKs where you own the code. +2. **What shape is it already in?** Structured JSON at the source, or unstructured + text needing grok/regex parsing? Parsing cost has to land *somewhere*. +3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate? +4. **What's the destination, and how many?** One backend forever, or several, or one + you expect to change? +5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent + on every host) distributes cost and adds resilience but multiplies config surface. + Parsing centrally concentrates cost and creates a single point of failure — but + one place to change rules. +6. **What already exists, and what does it cost?** Existing Elastic investment, + existing licences, team skills, and licensing constraints (Elastic's Filebeat and + Logstash are source-available under Elastic licensing; the OTel Collector, Fluent + Bit and Vector are permissively licensed). Migration is never free. + +### The answers + +**When to use Filebeat.** When you need a lightweight shipper that tails log *files* +from a host or container and forwards them with minimal transformation — and your +destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once +delivery with a registry of file offsets, backpressure handling, and prebuilt modules +for common services (nginx, MySQL, system logs) that come with parsing and dashboards +included. Reach for it when the source is a file you don't control the format of, the +app can't be modified, and Elasticsearch is where it's going. + +**When to use Logstash.** When you need real *transformation*: parsing unstructured +text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB), +splitting or merging events, conditional routing to multiple destinations, or +buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine +for events. It's heavyweight — a JVM — so you run a few of them centrally, not one +per host. + +**When to use Filebeat and Logstash together, and why.** This is the classic Elastic +architecture and the reason is a **division of labour between edge and centre**: + +- Filebeat runs on *every* host: cheap, tails files, handles offsets and + backpressure, ships onward. You want the thing running in a thousand places to be + small and boring. +- Logstash runs on a *few* central nodes: does the expensive grok parsing, + enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so + you want them in a handful of places you can redeploy, not on a thousand hosts. +- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's + persistent queue absorbs it and Filebeat backs off gracefully, rather than every + host queuing independently or dropping. + +So: Filebeat for *collection and delivery*, Logstash for *processing and routing*. +Use both when you have many hosts **and** non-trivial parsing/enrichment **and** +need central buffering. Skip Logstash when your logs are already structured JSON and +Elasticsearch ingest pipelines can do the little parsing that remains. + +**When to use OpenTelemetry.** + +- You own the application code and want telemetry generated *from inside it* — spans + with business context, custom metrics, logs already carrying trace IDs. +- You need **more than one signal**, correlated. This is the strongest argument: one + SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs + threaded through all three. Neither Filebeat nor Logstash does traces. +- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent + data model across all of them. +- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for + Jaeger by editing Collector YAML. +- You need distributed tracing at all. There is no serious vendor-neutral alternative. +- Your telemetry needs processing in flight — redaction, tail sampling, attribute + rewriting — without touching the app. + +**When *not* to use OpenTelemetry.** This is the question that separates people who +have used it from people who have read about it. + +- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary, + an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a + file-based collector. +- **You only need one signal from one source with no correlation.** A single nginx + access log going to a single log store: OTel is more machinery than the job needs. + Fluent Bit or Filebeat is simpler and finishes the job today. +- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature + Elastic estate with hundreds of Beats modules, dashboards and detection rules is + years of investment. "We should use OTel because it's the standard" is not a + business case. Note the direction of travel though: the Collector can already + receive from and export to most of these, so a *gradual* migration is usually the + right shape — often by putting a Collector or Vector in front of the existing + agents rather than ripping them out. +- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets, + where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes. +- **The signal is genuinely business analytics, not telemetry.** Clickstream data for + a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different + retention, different query patterns, different cost model. +- **Maturity gaps for a specific need in a specific language.** OTel logs support has + matured at different rates per language, and profiling is still alpha. Check the + status page for your language before betting on a signal. +- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks + on it. A two-person team that needs error tracking by Friday should install Sentry. + +The senior framing to give Babak: *these tools aren't competitors so much as +different layers.* OpenTelemetry is a **generation and transport standard** for +telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you +don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and +Elasticsearch are **stores**. Most real architectures use one from each layer, and +the interesting question is always which layer a given problem lives in. + +### Resources + +- Better Stack — *Filebeat vs Logstash: How to Choose* (free article). +- SigNoz — *OpenTelemetry vs ELK Stack* (free article). +- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* — + read these to see how the Collector does the Filebeat/Logstash job itself. +- OpenTelemetry docs → *Instrumentation status by language* — for the maturity + caveats. + +### Build — and this is no longer hypothetical + +Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml` +deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with +Filebeat reading Docker container logs and Logstash parsing JSON into daily +`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these +questions — the codebase he handed you contains both worlds. + +1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs** + flow down two pipelines: Filebeat → Logstash → Elasticsearch, and + Alloy → Loki. Compare them on the same data. +2. Test the direct ingest path the README documents — pipe a JSON line into + Logstash's TCP input on 5000, then query Elasticsearch — so you understand where + parsing happens before you change it. +3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter + that parses them. Then do the same job with Alloy's `loki.process` stages. Time + both. Note which one you'd rather debug at 2 a.m. +4. Compare the *investigation experience* directly: take one incident you already + diagnosed in Phase 9 and re-run it in Kibana Discover. What can you do there that + you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana + wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL + for some questions, and Loki's cost model genuinely beats Elasticsearch's for + others. +5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic + licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are + permissively licensed. For some employers that alone decides it. +6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most + valuably — what you actually found annoying about each. Having run both stacks + against the same application is the difference between reciting a comparison and + arguing one. diff --git a/docs/roadmap/phase-11.md b/docs/roadmap/phase-11.md new file mode 100644 index 0000000..d8550e2 --- /dev/null +++ b/docs/roadmap/phase-11.md @@ -0,0 +1,40 @@ +# Phase 11 — Package the delivery + +*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-10.md`](phase-10.md) · Next: [`summary.md`](summary.md)* + +**Status:** Not started + +**Message:** Depends on every prior phase being done — this is the wrap-up, +not independent work. + +**Time-box:** one evening. + +Babak gave you a task; deliver it like an engineer, not like a student. + +1. **`README.md`** — architecture diagram, one-command start, what each container is + for, and a "here's the demo" script. +2. **The five-minute demo path**, rehearsed: + - Delete every dashboard in the Grafana UI, then `ansible-playbook + playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt + from code, one command. + - Submit one transaction from the frontend, then follow it through Grafana: RED + dashboard → latency spike → exemplar → trace → span → logs for that span → back + to the trace. One request, all three signals, no searching. + - `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire, + and the service map turn red at a specific node. + - Show `docs/INVESTIGATION.md` and how you found the pathologies. +3. **The written answers** Babak asked for, as documents in the repo, not as a + conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`, + `docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99, + RTO/RPO/SLA/SLO/error budget in your own words. +4. **The test suite as part of the story.** Show `make test` green, then show the + three generations of the call-counting assertion side by side — + `docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what + each signal bought you. Most people demo dashboards; almost nobody can show + their observability work paying off inside their own test suite. +5. **A "what I'd do differently in production" section.** Tail sampling instead of + 100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet; + log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what + you *didn't* do, and why, is the strongest signal of seniority in the whole + delivery. + diff --git a/docs/roadmap/phase-2.md b/docs/roadmap/phase-2.md index 9f8451c..042c3ae 100644 --- a/docs/roadmap/phase-2.md +++ b/docs/roadmap/phase-2.md @@ -1,116 +1,89 @@ -# Phase 2 — Distributed tracing +# Phase 2 — The model, and one signal end to end *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)* **Status:** Not started -**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first. +**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is +described above. Waiting to begin. -**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task. +**Time-box:** one evening. -**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap -(Resource, providers, exporters, Collector wiring) is shared across all three -signals — do it once here and logs and metrics become incremental. Second, tracing -is what Babak actually asked for ("trace a single request, see where it stuck"), -and once trace IDs exist, log correlation in Phase 3 works on the first try instead -of needing rework. +**Why:** you need the vocabulary, but more importantly you need to see one trace +appear in Grafana within an hour, so the rest of the roadmap feels concrete rather +than abstract. ### Concepts -- **Trace / span / span context.** A trace is one request's journey; a span is one - timed operation within it. Each span carries a trace ID, its own span ID, and its - parent's span ID. That parent pointer is the entire data structure. -- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an - outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting - these right is what makes the service graph render correctly. -- **Context propagation.** - - *In-process*: OTel uses Python `contextvars`. This works across `await` — but - **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you - carry the context explicitly. - - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and - `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.) - - The classic bug: any boundary that doesn't carry those headers **starts a brand - new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with - `TraceContextTextMapPropagator().inject(...)` into the message payload on the - producer and `.extract(...)` on the consumer, then starting the consumer span - with that context as parent. This is the single most valuable exercise in the - phase — most people never do it manually and never really understand it. - - Related: `links` (for a consumer span that batches messages from many traces) - and **Baggage** (arbitrary key/values propagated alongside trace context — - useful, and a foot-gun, because it's sent on every hop and is visible to - downstream systems; never put secrets in it). -- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP - client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual - gets you the business semantics: `score_transaction`, `check_sanctions_list`, - `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces - that show *which service* was slow but not *why*. -- **Span attributes vs metric labels.** Spans are the right home for - high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale - with volume, not with distinct values. This is why "put it on the span, not the - metric" is the standing rule. -- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`. - That's how the exception ends up attached to the trace and how the `spanmetrics` - connector in Phase 4 will count your error rate. -- **Sampling — the concept now, the config later.** - - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops - errors at the same rate as successes. - - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow - traces, but must run in the Collector and buffer spans in memory. - - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off - now; you'll configure tail sampling in the Collector in Phase 5. - - Whatever you choose, the sampling decision must **propagate** — the sampled flag - rides in `traceparent`. Use `ParentBased` samplers so a service doesn't - unilaterally drop half of a trace its caller decided to keep. +- **Monitoring vs observability.** Monitoring answers questions you knew to ask + (known unknowns): dashboards and alerts built from predefined aggregates. + Observability is a property of the system: can you ask a *new* question about + production, that you never anticipated, without shipping new code? Charity Majors' + framing — monitoring is known-unknowns, observability is unknown-unknowns. The + practical implication for you: emit **wide, high-dimensional events** (rich span + attributes), not just pre-aggregated counters. +- **The OTel architecture split: API vs SDK vs Collector.** + - The **API** is what your code and third-party instrumentation libraries call. + If no SDK is configured, API calls are no-ops — that's deliberate, it means a + library can be instrumented without forcing a dependency on any implementation. + - The **SDK** is the engine: sampling, batching, context propagation, export. + Application owners configure the SDK; library authors only touch the API. + - The **Collector** is a standalone process that receives, processes and exports + telemetry. Optional, but you should use one. +- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318. +- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`, + `service.version`, `deployment.environment`) and is attached to every signal from + that process. Attributes describe the individual span/metric/log. +- **Semantic conventions**: standardised attribute names (`http.request.method`, + `db.system`, `server.address`). Use them for anything standard; namespace your own + (`ledgerline.account.id`). This is what makes cross-service, cross-language + correlation work — and it's what Grafana's prebuilt dashboards key off. +- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha — + aware of, not learning yet.) +- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your + app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of + seven services. The Collector also does the things you don't want in your app: + retry, PII redaction, tail sampling, memory limiting, fan-out to multiple + destinations. ### Resources -- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30 - min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both. -- Your chosen Udemy backbone: the instrumentation + context-propagation sections. -- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most - practical FastAPI-specific setup, including batch/queue env vars. -- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not - reading. -- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services - side by side. Look specifically at how they mix auto-instrumentation with - hand-rolled business spans, and how the Kafka producer/consumer propagate context. - This is the "professional implementation" you asked to study. +- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the + M.E.L.T. and context-propagation sections; skip the Node code. +- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min. +- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop + repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand + it — just see what "done" looks like. ### Build -1. **Shared bootstrap.** Create `common/telemetry.py` with a single - `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`, - `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every - service calls it once at startup. Keep it under 60 lines. -2. **Auto-instrument all seven services**: `FastAPIInstrumentor`, - `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify - you now get a connected trace across `gateway → ledger → fraud → sanctions`. -3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`, - `sanctions.lookup_watchlist`, `insights.aggregate_120d`, - `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and - namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`, - `ledgerline.tx.amount_minor`). -4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span, - inject trace context into the message fields. Consumers (`notifier`, `audit`): - extract, and start a `CONSUMER` span with the extracted context as parent. -5. **Record exceptions** everywhere you currently `logger.exception(...)`. -6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly - more. -7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded - the "before" state so you could measure against it, and from here it's - actively wrong. Everything else in `tests/` must still pass: instrumenting - the system is not allowed to change its behaviour, and if `make test` goes red - after adding spans, you've broken something real. That is exactly the safety - net you built it for. +1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it, + `ansible-galaxy collection install -r requirements.yml`, change + `grafana_admin_password` (vault it), then + `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the + README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093, + Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318. + Then open the Alloy UI and **read the component graph** — it shows you the exact + pipeline, including the fact that the OTLP receiver has only a traces output. + Seeing that in the UI before you get bitten by it is worth ten minutes. + *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`, + a single container with everything pre-wired. Useful for a first trace, but + don't build on it — its pipeline differs from your real platform in exactly the + ways that matter.)* +2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code + auto-instrumentation: + ``` + pip install opentelemetry-distro opentelemetry-exporter-otlp + opentelemetry-bootstrap -a install + ``` + and change its command to + `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003` + with `OTEL_SERVICE_NAME=sanctions`, + `OTEL_EXPORTER_OTLP_ENDPOINT=http://:4318`, + `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at + Tempo directly. +3. Fire a request. Open Grafana → Explore → Tempo → find the trace. -### Done when - -- One request produces **one connected trace** covering all seven services, - including both async consumers. -- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that - ate the time. -- You can explain why `blocking_event_loop` looks different in a trace than - `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap* - before spans start, plus every concurrent request in that process degrading - together.) +**Done when:** you can see a single-service trace in Tempo, and you can explain out +loud what the Resource attributes on it mean and where they came from. diff --git a/docs/roadmap/phase-3.md b/docs/roadmap/phase-3.md index 98291bd..59c28bc 100644 --- a/docs/roadmap/phase-3.md +++ b/docs/roadmap/phase-3.md @@ -1,99 +1,116 @@ -# Phase 3 — Structured logs, correlated to traces +# Phase 3 — Distributed tracing *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)* **Status:** Not started -**Message:** Depends on Phase 2 (trace IDs must exist before log correlation -means anything). +**Message:** Depends on Phase 2's `common/telemetry.py` bootstrap existing first. -**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win. +**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task. + +**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap +(Resource, providers, exporters, Collector wiring) is shared across all three +signals — do it once here and logs and metrics become incremental. Second, tracing +is what Babak actually asked for ("trace a single request, see where it stuck"), +and once trace IDs exist, log correlation in Phase 4 works on the first try instead +of needing rework. ### Concepts -- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with - events: a stable event name plus typed fields (`event="transaction_settled"`, - `account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names. - Consistent field names across all seven services — `account_id` everywhere, never - `accountId` in one place and `acct` in another. This is what makes LogQL queries - possible at all. -- **Trace–log correlation.** Every log line emitted inside an active span must carry - `trace_id` and `span_id`. This is what turns "I have logs and I have traces" into - "one pane of glass": click a span → see its logs; click a log line → open its - trace. Both directions are configured, and you'll set both up. -- **Two ways to ship logs, and which one your platform actually does:** - - **stdout → collector scraping → Loki.** The container-native way. Your app just - prints JSON; the pipeline handles the rest. Decoupled, survives app crashes, - standard in Kubernetes. **This is the only path wired in your Alloy config** - (`loki.source.docker` reading the Docker socket). - - **OTLP log exporter → collector → Loki.** Logs travel the same path as traces, - with the same Resource attributes attached automatically. Cleaner correlation, - but the app owns delivery. **On your platform this currently goes nowhere** — - Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and - dropped in silence. - - Do both anyway: use stdout for most services, and *extend Alloy* to wire the - OTLP logs path for one or two. Then argue the trade-off from experience. That - comparison — including the fact that a misconfigured pipeline fails silently - rather than loudly — is precisely what Babak will probe. -- **The field name is not a style choice here.** Your Loki datasource is provisioned - with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive. - A log line must carry lowercase `trace_id`. OpenTelemetry's logging - auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get - logs and traces both arriving, no errors anywhere, and no clickable link. Hence - structlog with an explicit processor rather than the auto-injection shortcut. -- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce - your entire fleet, and Loki bills/indexes on volume. -- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive - `json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build - redaction into the processor chain, not into each call site. -- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*, - not log content. Labels should be few and low-cardinality (`service`, `env`, - `level`). Everything else lives in the log line body and is filtered at query time. - Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it - in the JSON body and use a **derived field** to make it clickable. +- **Trace / span / span context.** A trace is one request's journey; a span is one + timed operation within it. Each span carries a trace ID, its own span ID, and its + parent's span ID. That parent pointer is the entire data structure. +- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an + outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting + these right is what makes the service graph render correctly. +- **Context propagation.** + - *In-process*: OTel uses Python `contextvars`. This works across `await` — but + **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you + carry the context explicitly. + - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and + `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.) + - The classic bug: any boundary that doesn't carry those headers **starts a brand + new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with + `TraceContextTextMapPropagator().inject(...)` into the message payload on the + producer and `.extract(...)` on the consumer, then starting the consumer span + with that context as parent. This is the single most valuable exercise in the + phase — most people never do it manually and never really understand it. + - Related: `links` (for a consumer span that batches messages from many traces) + and **Baggage** (arbitrary key/values propagated alongside trace context — + useful, and a foot-gun, because it's sent on every hop and is visible to + downstream systems; never put secrets in it). +- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP + client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual + gets you the business semantics: `score_transaction`, `check_sanctions_list`, + `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces + that show *which service* was slow but not *why*. +- **Span attributes vs metric labels.** Spans are the right home for + high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale + with volume, not with distinct values. This is why "put it on the span, not the + metric" is the standing rule. +- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`. + That's how the exception ends up attached to the trace and how the `spanmetrics` + connector in Phase 5 will count your error rate. +- **Sampling — the concept now, the config later.** + - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops + errors at the same rate as successes. + - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow + traces, but must run in the Collector and buffer spans in memory. + - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off + now; you'll configure tail sampling in the Collector in Phase 6. + - Whatever you choose, the sampling decision must **propagate** — the sampled flag + rides in `traceparent`. Use `ParentBased` samplers so a service doesn't + unilaterally drop half of a trace its caller decided to keep. ### Resources -- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and - Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor - setup, including the hex formatting for trace/span IDs. -- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for - `OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID` - fields, if you'd rather not hand-roll a processor. -- Udemy backbone: the Loki + LogQL sections. -- Grafana docs → Loki → *Send data via OTLP* and *derived fields*. +- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30 + min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both. +- Your chosen Udemy backbone: the instrumentation + context-propagation sections. +- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most + practical FastAPI-specific setup, including batch/queue env vars. +- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not + reading. +- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services + side by side. Look specifically at how they mix auto-instrumentation with + hand-rolled business spans, and how the Kafka producer/consumer propagate context. + This is the "professional implementation" you asked to study. ### Build -1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a - processor that pulls the current span context and injects `trace_id` / `span_id`. -2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per - handler: request received (or rely on the span), business decision, error. -3. Add a redaction processor with a denylist of field names. -4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki - with a `container` label (Alloy's relabel rule sets it from the Docker container - name). Verify the `trace_id` field is present *and lowercase* in the JSON body. -5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service, - send some records, and watch nothing appear in Loki. Then check the Alloy UI: - the receiver shows traffic, and the pipeline shows no logs output. Now extend - `config.alloy.j2` — add a `logs` output on the receiver plus an - `otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch - them arrive. Write down what you'd have done to debug this if you *hadn't* known. - That is the most transferable half-hour in the whole project. -6. Confirm the provisioned correlation works in both directions: Loki's - `derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2` - (`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you - are verifying, not writing. If a link doesn't appear, the regex didn't match - your field name — check that before anything else. -6. **Migrate the test suite's log-counting helpers.** Three pathology tests - (`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines - in `docker compose logs` — the only cross-service call counter that existed at - Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki - with LogQL over structured fields instead. This is a small change and a - pointed one: notice how much shorter and more precise the assertion gets, and - that it now works without shelling out to Docker at all. That delta is the - value of structured logging, measured on your own code. +1. **Shared bootstrap.** Create `common/telemetry.py` with a single + `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`, + `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every + service calls it once at startup. Keep it under 60 lines. +2. **Auto-instrument all seven services**: `FastAPIInstrumentor`, + `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify + you now get a connected trace across `gateway → ledger → fraud → sanctions`. +3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`, + `sanctions.lookup_watchlist`, `insights.aggregate_120d`, + `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and + namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`, + `ledgerline.tx.amount_minor`). +4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span, + inject trace context into the message fields. Consumers (`notifier`, `audit`): + extract, and start a `CONSUMER` span with the extracted context as parent. +5. **Record exceptions** everywhere you currently `logger.exception(...)`. +6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly + more. +7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded + the "before" state so you could measure against it, and from here it's + actively wrong. Everything else in `tests/` must still pass: instrumenting + the system is not allowed to change its behaviour, and if `make test` goes red + after adding spans, you've broken something real. That is exactly the safety + net you built it for. -**Done when:** from a slow trace you can click into that request's logs, and from -any log line you can click into its trace — in both directions, in two clicks. +### Done when + +- One request produces **one connected trace** covering all seven services, + including both async consumers. +- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that + ate the time. +- You can explain why `blocking_event_loop` looks different in a trace than + `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap* + before spans start, plus every concurrent request in that process degrading + together.) diff --git a/docs/roadmap/phase-4.md b/docs/roadmap/phase-4.md index 4d884b7..631b343 100644 --- a/docs/roadmap/phase-4.md +++ b/docs/roadmap/phase-4.md @@ -1,126 +1,99 @@ -# Phase 4 — Metrics +# Phase 4 — Structured logs, correlated to traces *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)* **Status:** Not started -**Message:** Not blocked technically (metrics don't strictly need tracing -first), but sequenced after Phase 3 per the roadmap's ordering. +**Message:** Depends on Phase 3 (trace IDs must exist before log correlation +means anything). -**Time-box:** 2 evenings. +**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win. ### Concepts -- **Instrument types.** - - **Counter** — monotonic, only increases. Requests, errors, messages consumed. - - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests, - connection-pool checkouts. - - **Histogram** — a distribution. Request latency, payload size. This is what - gives you percentiles. - - Synchronous instruments are recorded inline; **observable/asynchronous** ones are - read via a callback on collection (right for things you *poll*, like pool size). - - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore - **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic - nonsense. Use histograms. -- **Cardinality — the one thing that will bite you.** Every unique combination of - metric name + label values is a separate time series in memory. Add `account_id` - as a label on a 800-account demo and you've made 800 series; do it in production - and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed - ~10 000 distinct values in production, it does not belong on a metric.** Put it on - a span. Histograms multiply this (one series per bucket per label combination), so - don't over-bucket either. - - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not - the concrete path), `http.status_code` class, `kind`, `currency`. - - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`, - raw URL, timestamp, error message. -- **RED and USE.** - - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your - per-service dashboard. - - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory, - connection pools, queues, disk. Saturation is the one everyone forgets and the - one that predicts collapse. - - You want both: RED tells you the service is slow, USE tells you which resource - is why. -- **Span metrics — generate metrics from traces.** Two architectures exist for this, - and knowing both is the point: - - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and - `servicegraph` connectors read spans in flight and emit RED metrics and edge - metrics. This is what most blog posts (including the OneUptime article) show. - - *In the trace store:* Tempo's **metrics-generator** does the same job after - ingestion, remote-writing the result to Prometheus. **This is what your platform - does** — `processors: [service-graphs, span-metrics, local-blocks]` with - `send_exemplars: true`, remote-writing to a Prometheus started with - `--web.enable-remote-write-receiver`. - - So RED metrics, the service map and exemplars **already exist** before you write - a single metric line. Do not add the collector connectors as well — you'd - duplicate every series under a second set of names. **Go find the real metric - names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL; - they will not match the article's. - - The trade-off worth being able to argue: collector-side generation is cheaper - and works before storage, but sees only what passes through that collector; - Tempo-side generation sees everything that landed, survives collector restarts, - and can be enabled per-tenant — at the cost of load on the trace store. -- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you - click a spike on a latency graph and land in the exact trace. On your platform - this is already wired end to end: Tempo's generator sets `send_exemplars: true`, - and the Prometheus datasource declares `exemplarTraceIdDestinations` with - `name: trace_id`. If you emit your own histograms with exemplars, the label must - be `trace_id` to match. -- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you - can do either — receive OTLP from apps, then either expose a `/metrics` endpoint - for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that - the pull model is why Prometheus has service discovery and why `up` exists. +- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with + events: a stable event name plus typed fields (`event="transaction_settled"`, + `account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names. + Consistent field names across all seven services — `account_id` everywhere, never + `accountId` in one place and `acct` in another. This is what makes LogQL queries + possible at all. +- **Trace–log correlation.** Every log line emitted inside an active span must carry + `trace_id` and `span_id`. This is what turns "I have logs and I have traces" into + "one pane of glass": click a span → see its logs; click a log line → open its + trace. Both directions are configured, and you'll set both up. +- **Two ways to ship logs, and which one your platform actually does:** + - **stdout → collector scraping → Loki.** The container-native way. Your app just + prints JSON; the pipeline handles the rest. Decoupled, survives app crashes, + standard in Kubernetes. **This is the only path wired in your Alloy config** + (`loki.source.docker` reading the Docker socket). + - **OTLP log exporter → collector → Loki.** Logs travel the same path as traces, + with the same Resource attributes attached automatically. Cleaner correlation, + but the app owns delivery. **On your platform this currently goes nowhere** — + Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and + dropped in silence. + - Do both anyway: use stdout for most services, and *extend Alloy* to wire the + OTLP logs path for one or two. Then argue the trade-off from experience. That + comparison — including the fact that a misconfigured pipeline fails silently + rather than loudly — is precisely what Babak will probe. +- **The field name is not a style choice here.** Your Loki datasource is provisioned + with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive. + A log line must carry lowercase `trace_id`. OpenTelemetry's logging + auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get + logs and traces both arriving, no errors anywhere, and no clickable link. Hence + structlog with an explicit processor rather than the auto-injection shortcut. +- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce + your entire fleet, and Loki bills/indexes on volume. +- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive + `json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build + redaction into the processor chain, not into each call site. +- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*, + not log content. Labels should be few and low-cardinality (`service`, `env`, + `level`). Everything else lives in the log line body and is filtered at query time. + Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it + in the JSON body and use a **derived field** to make it clickable. ### Resources -- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min). - Read the first half. Best cardinality explanation available. -- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill, - not a fact. -- **The OneUptime APM article** — now it's time. Take its `spanmetrics` / - `servicegraph` connector config and its RED PromQL queries almost verbatim. -- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics. +- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and + Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor + setup, including the hex formatting for trace/span IDs. +- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for + `OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID` + fields, if you'd rather not hand-roll a processor. +- Udemy backbone: the Loki + LogQL sections. +- Grafana docs → Loki → *Send data via OTLP* and *derived fields*. ### Build -1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter. -2. Custom application metrics, deliberately few: - - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`) - - `ledgerline.transaction.duration` (Histogram; labels: `kind`) - - `ledgerline.fraud.score` (Histogram; no labels) - - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`) - - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`) -3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP - metrics, so pick a path and implement it: - - *Platform grain:* expose `/metrics` on each service and add them to - `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the - playbook. Prometheus scrapes your services directly. Note the irony and the - lesson — you're back to a pull model and a second client library, which is - exactly the fragmentation OTel exists to fix. - - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to - Alloy so app metrics flow through the collector as the architecture drawing - intends. This also closes a real diagram-vs-code gap in the repo. - - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire - `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for - all three signals. - Do the first to get moving; do one of the others afterwards and write up which - you'd choose for a real platform team and why. That write-up is a Phase 9 - artefact. -4. Deliberately break it: add `account_id` as a label to - `ledgerline.transactions.total`, run a 200-request burst, and watch the series - count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now - you'll never forget. -5. Compare your hand-written duration histogram against the `spanmetrics`-generated - one. Understand why they differ slightly and when you'd want each. -6. **Second migration of the call-counting assertions.** The three tests you moved - to LogQL in Phase 3 can now count **client spans** in Tempo instead — a - TraceQL query for `sanctions` client spans within a single trace answers - "how many lookups did this one transaction make?" directly, which is the - question the test was always really asking. Do it and compare all three - generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the - clearest possible demonstration of what each signal is actually for, on a - problem you already understand. +1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a + processor that pulls the current span context and injects `trace_id` / `span_id`. +2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per + handler: request received (or rely on the span), business decision, error. +3. Add a redaction processor with a denylist of field names. +4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki + with a `container` label (Alloy's relabel rule sets it from the Docker container + name). Verify the `trace_id` field is present *and lowercase* in the JSON body. +5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service, + send some records, and watch nothing appear in Loki. Then check the Alloy UI: + the receiver shows traffic, and the pipeline shows no logs output. Now extend + `config.alloy.j2` — add a `logs` output on the receiver plus an + `otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch + them arrive. Write down what you'd have done to debug this if you *hadn't* known. + That is the most transferable half-hour in the whole project. +6. Confirm the provisioned correlation works in both directions: Loki's + `derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2` + (`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you + are verifying, not writing. If a link doesn't appear, the regex didn't match + your field name — check that before anything else. +6. **Migrate the test suite's log-counting helpers.** Three pathology tests + (`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines + in `docker compose logs` — the only cross-service call counter that existed at + Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki + with LogQL over structured fields instead. This is a small change and a + pointed one: notice how much shorter and more precise the assertion gets, and + that it now works without shelling out to Docker at all. That delta is the + value of structured logging, measured on your own code. -**Done when:** you have a RED view per service, a working service map, and you can -click a latency spike through to a trace via an exemplar. +**Done when:** from a slow trace you can click into that request's logs, and from +any log line you can click into its trace — in both directions, in two clicks. diff --git a/docs/roadmap/phase-5.md b/docs/roadmap/phase-5.md index 9bbfc1b..8ee9adf 100644 --- a/docs/roadmap/phase-5.md +++ b/docs/roadmap/phase-5.md @@ -1,76 +1,126 @@ -# Phase 5 — Dashboards and alerts as code +# Phase 5 — Metrics *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)* **Status:** Not started -**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth -dashboarding. +**Message:** Not blocked technically (metrics don't strictly need tracing +first), but sequenced after Phase 4 per the roadmap's ordering. -**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and -recreate in seconds.* +**Time-box:** 2 evenings. ### Concepts -- Three levels of "as code", and you should know when each applies: - 1. **File provisioning** — YAML in `provisioning/datasources/` and - `provisioning/dashboards/`, plus dashboard JSON files mounted into the - container. Grafana loads them at startup. Simplest, no state, no API tokens, - perfect for a Compose demo. **This alone satisfies the requirement.** - 2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java, - PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a - function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable - diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead - of Grafonnet (which is legacy and unofficial). - 3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed, - multi-environment, or GitOps setups. Overkill here; the right answer *later*, - when this lands on your Kubernetes cluster. -- **Dashboards should answer questions, not display data.** A good service dashboard - answers, in order: is it up, is it serving, is it erroring, is it slow, and where - is the pressure. Panels that nobody looks at during an incident are cost, not value. -- **Alerts as code too.** Grafana alert rules, contact points and notification - policies all provision from YAML. And keep the *alert definition* next to the - dashboard that explains it. +- **Instrument types.** + - **Counter** — monotonic, only increases. Requests, errors, messages consumed. + - **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests, + connection-pool checkouts. + - **Histogram** — a distribution. Request latency, payload size. This is what + gives you percentiles. + - Synchronous instruments are recorded inline; **observable/asynchronous** ones are + read via a callback on collection (right for things you *poll*, like pool size). + - Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore + **cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic + nonsense. Use histograms. +- **Cardinality — the one thing that will bite you.** Every unique combination of + metric name + label values is a separate time series in memory. Add `account_id` + as a label on a 800-account demo and you've made 800 series; do it in production + and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed + ~10 000 distinct values in production, it does not belong on a metric.** Put it on + a span. Histograms multiply this (one series per bucket per label combination), so + don't over-bucket either. + - Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not + the concrete path), `http.status_code` class, `kind`, `currency`. + - Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`, + raw URL, timestamp, error message. +- **RED and USE.** + - **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your + per-service dashboard. + - **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory, + connection pools, queues, disk. Saturation is the one everyone forgets and the + one that predicts collapse. + - You want both: RED tells you the service is slow, USE tells you which resource + is why. +- **Span metrics — generate metrics from traces.** Two architectures exist for this, + and knowing both is the point: + - *In the collector:* the OpenTelemetry Collector's `spanmetrics` and + `servicegraph` connectors read spans in flight and emit RED metrics and edge + metrics. This is what most blog posts (including the OneUptime article) show. + - *In the trace store:* Tempo's **metrics-generator** does the same job after + ingestion, remote-writing the result to Prometheus. **This is what your platform + does** — `processors: [service-graphs, span-metrics, local-blocks]` with + `send_exemplars: true`, remote-writing to a Prometheus started with + `--web.enable-remote-write-receiver`. + - So RED metrics, the service map and exemplars **already exist** before you write + a single metric line. Do not add the collector connectors as well — you'd + duplicate every series under a second set of names. **Go find the real metric + names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL; + they will not match the article's. + - The trade-off worth being able to argue: collector-side generation is cheaper + and works before storage, but sees only what passes through that collector; + Tempo-side generation sees everything that landed, survives collector restarts, + and can be enabled per-tenant — at the cost of load on the trace store. +- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you + click a spike on a latency graph and land in the exact trace. On your platform + this is already wired end to end: Tempo's generator sets `send_exemplars: true`, + and the Prometheus datasource declares `exemplarTraceIdDestinations` with + `name: trace_id`. If you emit your own histograms with exemplars, the label must + be `trace_id` to match. +- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you + can do either — receive OTLP from apps, then either expose a `/metrics` endpoint + for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that + the pull model is why Prometheus has service discovery and why `up` exists. ### Resources -- Grafana docs → *Observability as code* (the index page — it lays out Foundation - SDK, file provisioning, Git Sync and Terraform, and tells you which to use when). -- Grafana Foundation SDK repo → the Python examples, especially the `red-method` - example. -- Udemy backbone: dashboard design sections (panel types, transformations, - variables, dashboard best practice). +- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min). + Read the first half. Best cardinality explanation available. +- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill, + not a fact. +- **The OneUptime APM article** — now it's time. Take its `spanmetrics` / + `servicegraph` connector config and its RED PromQL queries almost verbatim. +- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics. ### Build -Your platform provisions **datasources only** — there is no dashboards provider at -all. So this phase is entirely yours, and the right home for it is inside the -Ansible role, using the same mechanism as everything else. +1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter. +2. Custom application metrics, deliberately few: + - `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`) + - `ledgerline.transaction.duration` (Histogram; labels: `kind`) + - `ledgerline.fraud.score` (Histogram; no labels) + - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`) + - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`) +3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP + metrics, so pick a path and implement it: + - *Platform grain:* expose `/metrics` on each service and add them to + `prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the + playbook. Prometheus scrapes your services directly. Note the irony and the + lesson — you're back to a pull model and a second client library, which is + exactly the fragmentation OTel exists to fix. + - *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to + Alloy so app metrics flow through the collector as the architecture drawing + intends. This also closes a real diagram-vs-code gap in the repo. + - *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire + `otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for + all three signals. + Do the first to get moving; do one of the others afterwards and write up which + you'd choose for a real platform team and why. That write-up is a Phase 10 + artefact. +4. Deliberately break it: add `account_id` as a label to + `ledgerline.transactions.total`, run a 200-request burst, and watch the series + count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now + you'll never forget. +5. Compare your hand-written duration histogram against the `spanmetrics`-generated + one. Understand why they differ slightly and when you'd want each. +6. **Second migration of the call-counting assertions.** The three tests you moved + to LogQL in Phase 4 can now count **client spans** in Tempo instead — a + TraceQL query for `sanctions` client spans within a single trace answers + "how many lookups did this one transaction make?" directly, which is the + question the test was always really asking. Do it and compare all three + generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the + clearest possible demonstration of what each signal is actually for, on a + problem you already understand. -1. **Extend the role.** Add - `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2` - (a file-based dashboard provider pointing at a mounted directory), a - `config/grafana/dashboards/` directory for the JSON, and the tasks to template - and copy them. The Compose template already mounts - `./config/grafana/provisioning` read-only, so the provider is picked up on - restart. -2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana - Foundation SDK (Python)**: one function producing a RED dashboard for a given - service name, looped over the seven services; a second producing a system - overview (service graph, stream lag, pool saturation, error-budget burn). Commit - the generator *and* its output; the Ansible role ships the output. -3. **The demo is now `ansible-playbook`.** Delete every dashboard through the - Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back. - That is a strictly stronger answer to Babak's "destroy and recreate in seconds" - than a local Makefile target, because it's the same command that builds the - platform. -4. While you're in there, review the provisioned datasources and add what's missing: - `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a - worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping — - understand why `filterByTraceID: true` alone is sufficient here, and when it - wouldn't be. -5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice. - -**Done when:** re-running the playbook restores every dashboard from scratch, and no -dashboard has ever been saved through the Grafana UI. +**Done when:** you have a RED view per service, a working service map, and you can +click a latency spike through to a trace via an exemplar. diff --git a/docs/roadmap/phase-6.md b/docs/roadmap/phase-6.md index 8bf66f3..8738d19 100644 --- a/docs/roadmap/phase-6.md +++ b/docs/roadmap/phase-6.md @@ -1,94 +1,76 @@ -# Phase 6 — Load testing with k6, and percentiles +# Phase 6 — Dashboards and alerts as code *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)* **Status:** Not started -**Message:** Depends on Phase 4 (metrics to plot k6's output against) and -benefits from Phase 5's dashboards existing. +**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth +dashboarding. -**Time-box:** 1–2 evenings. +**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and +recreate in seconds.* -### Concepts — P95 and P99 properly +### Concepts -You asked about these specifically, and there's more to them than "the 95th -percentile." - -- **A percentile is meaningless without a population and a window.** "p95 = 400 ms" - is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms" - is. -- **You cannot average percentiles.** Not across instances, not across time buckets. - `avg(p99)` is wrong and produces a number that looks plausible. The correct move is - to aggregate the *histogram buckets* and compute the quantile from the sum: - `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`. -- **Prometheus classic histograms give you an interpolated estimate**, bounded by - your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that - bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that - bracket your actual SLO threshold. (Native histograms largely solve this and are - worth knowing about.) -- **Which population?** An overall p99 across all endpoints is dominated by whichever - endpoint has the most traffic. Slice by route, or you'll miss that - `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine. -- **Tail latency amplification** — the most important idea here for a microservice - system. If one user request fans out to 10 backend calls, each with an independent - p99 of 1 s, then the chance the *user request* touches at least one slow call is - `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's - p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is - fine" and "the product is slow" coexist. Ledgerline's fan-out makes this - measurable — go measure it. -- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable, - most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly - that. -- **Coordinated omission** — a load-generator trap. If your generator waits for a - response before sending the next request, then when the system stalls, the - generator stalls too and simply *doesn't record* the requests that would have been - slow. Your measured latency is systematically optimistic. Fix: use an **open - model** — a fixed *arrival rate* independent of response times. In k6 that means - the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than - `constant-vus`. Knowing this is a genuine senior-level differentiator. - -### Concepts — k6 - -- **Test types:** smoke (does it work at all), average-load (normal traffic), - stress (past normal, find the knee), spike (sudden surge, test recovery), - soak (hours, find leaks and lag drift), breakpoint (ramp until failure). - Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak. -- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model - (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above. -- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't - fail the run; **thresholds are SLOs expressed in code and they set the process exit - code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`. -- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's - view and the application's view sit on the same Grafana time axis. That side-by-side - is the single most convincing screenshot you can show Babak. -- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're - writing 60 lines of it. +- Three levels of "as code", and you should know when each applies: + 1. **File provisioning** — YAML in `provisioning/datasources/` and + `provisioning/dashboards/`, plus dashboard JSON files mounted into the + container. Grafana loads them at startup. Simplest, no state, no API tokens, + perfect for a Compose demo. **This alone satisfies the requirement.** + 2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java, + PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a + function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable + diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead + of Grafonnet (which is legacy and unofficial). + 3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed, + multi-environment, or GitOps setups. Overkill here; the right answer *later*, + when this lands on your Kubernetes cluster. +- **Dashboards should answer questions, not display data.** A good service dashboard + answers, in order: is it up, is it serving, is it erroring, is it slow, and where + is the pressure. Panels that nobody looks at during an incident are cost, not value. +- **Alerts as code too.** Grafana alert rules, contact points and notification + policies all provision from YAML. And keep the *alert definition* next to the + dashboard that explains it. ### Resources -- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*, - *Thresholds*. Free, genuinely well written; ~2 h to work through. -- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free, - hands-on). -- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one. -- Grafana docs → *k6 output to Prometheus remote write*. +- Grafana docs → *Observability as code* (the index page — it lays out Foundation + SDK, file provisioning, Git Sync and Terraform, and tells you which to use when). +- Grafana Foundation SDK repo → the Python examples, especially the `red-method` + example. +- Udemy backbone: dashboard design sections (panel types, transformations, + variables, dashboard best practice). ### Build -1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`, - sharing a `common.js` for setup and payload generation against random accounts. -2. Use **arrival-rate executors** for `average` and `stress`. Write down why. -3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`, - `http_req_failed rate<0.01`. -4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard. -5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result - sets. This before/after pair is your baseline for Phase 8. -6. Optional but impressive: have k6 send W3C trace context headers so load-test - requests appear as traces, letting you click from a k6 spike into a specific - trace. Check the current k6 docs for the tracing module's status — it has moved - between experimental and stable. +Your platform provisions **datasources only** — there is no dashboards provider at +all. So this phase is entirely yours, and the right home for it is inside the +Ansible role, using the same mechanism as everything else. -**Done when:** `make load-stress` produces a Grafana view where you can see the -arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the -service that broke first. +1. **Extend the role.** Add + `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2` + (a file-based dashboard provider pointing at a mounted directory), a + `config/grafana/dashboards/` directory for the JSON, and the tasks to template + and copy them. The Compose template already mounts + `./config/grafana/provisioning` read-only, so the provider is picked up on + restart. +2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana + Foundation SDK (Python)**: one function producing a RED dashboard for a given + service name, looped over the seven services; a second producing a system + overview (service graph, stream lag, pool saturation, error-budget burn). Commit + the generator *and* its output; the Ansible role ships the output. +3. **The demo is now `ansible-playbook`.** Delete every dashboard through the + Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back. + That is a strictly stronger answer to Babak's "destroy and recreate in seconds" + than a local Makefile target, because it's the same command that builds the + platform. +4. While you're in there, review the provisioned datasources and add what's missing: + `tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a + worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping — + understand why `filterByTraceID: true` alone is sufficient here, and when it + wouldn't be. +5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice. + +**Done when:** re-running the playbook restores every dashboard from scratch, and no +dashboard has ever been saved through the Grafana UI. diff --git a/docs/roadmap/phase-7.md b/docs/roadmap/phase-7.md index cab7cc8..ce121cd 100644 --- a/docs/roadmap/phase-7.md +++ b/docs/roadmap/phase-7.md @@ -1,91 +1,94 @@ -# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO +# Phase 7 — Load testing with k6, and percentiles *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)* **Status:** Not started -**Message:** Depends on Phase 6's load-test baseline (the latency SLO should -come from measured numbers, not be picked from thin air). +**Message:** Depends on Phase 5 (metrics to plot k6's output against) and +benefits from Phase 6's dashboards existing. -**Time-box:** 1–2 evenings. Mostly thinking, a little config. +**Time-box:** 1–2 evenings. -### Concepts +### Concepts — P95 and P99 properly -- **SLI** — a *measurement* of something users experience, expressed as a ratio of - good events to valid events. Not "CPU is at 60%". Rather: *the proportion of - `POST /api/transactions` requests that returned 2xx within 500 ms.* -- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction - requests succeed within 500 ms, measured over a rolling 28 days." -- **SLA** — an external, *contractual* promise, with consequences (refunds, credits). - Always **looser** than your SLO, so you have room to miss internally without - breaching commercially. If your SLA and SLO are the same number, you've made a - mistake. -- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1% - budget; over 30 days that's about **43 minutes**. The budget is the thing that - turns reliability from an argument into arithmetic: if there's budget left, ship - features and take risks; if it's spent, the policy says you stop feature work and - spend on reliability. That policy — agreed in advance by dev, ops and product — is - the whole point. -- **Burn rate.** How fast you're consuming budget relative to the window. - Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's - multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket) - is what stops you from either paging on every blip or noticing the slow bleed a - month late. -- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs, - and it's worth being precise because interviewers probe exactly here. - - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long - from failure to restored service. It drives your architecture: replication, - warm standby, failover automation, runbook quality. - - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured - in time: how far back can you afford to lose. It drives your **backup and - replication frequency**. RPO of 15 minutes means backups/WAL shipping at least - every 15 minutes. - - The relationship worth stating out loud: **SLO governs steady-state quality; RTO - and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says - "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most - 5 minutes of transactions." Both feed the SLA. - - Both are **business decisions, not technical ones** — you derive them by asking - what an hour of downtime or an hour of lost data actually costs, then buying the - cheapest architecture that meets it. Tightening either is expensive and - non-linear. +You asked about these specifically, and there's more to them than "the 95th +percentile." + +- **A percentile is meaningless without a population and a window.** "p95 = 400 ms" + is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms" + is. +- **You cannot average percentiles.** Not across instances, not across time buckets. + `avg(p99)` is wrong and produces a number that looks plausible. The correct move is + to aggregate the *histogram buckets* and compute the quantile from the sum: + `histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`. +- **Prometheus classic histograms give you an interpolated estimate**, bounded by + your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that + bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that + bracket your actual SLO threshold. (Native histograms largely solve this and are + worth knowing about.) +- **Which population?** An overall p99 across all endpoints is dominated by whichever + endpoint has the most traffic. Slice by route, or you'll miss that + `/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine. +- **Tail latency amplification** — the most important idea here for a microservice + system. If one user request fans out to 10 backend calls, each with an independent + p99 of 1 s, then the chance the *user request* touches at least one slow call is + `1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's + p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is + fine" and "the product is slow" coexist. Ledgerline's fan-out makes this + measurable — go measure it. +- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable, + most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly + that. +- **Coordinated omission** — a load-generator trap. If your generator waits for a + response before sending the next request, then when the system stalls, the + generator stalls too and simply *doesn't record* the requests that would have been + slow. Your measured latency is systematically optimistic. Fix: use an **open + model** — a fixed *arrival rate* independent of response times. In k6 that means + the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than + `constant-vus`. Knowing this is a genuine senior-level differentiator. + +### Concepts — k6 + +- **Test types:** smoke (does it work at all), average-load (normal traffic), + stress (past normal, find the knee), spike (sudden surge, test recovery), + soak (hours, find leaks and lag drift), breakpoint (ramp until failure). + Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak. +- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model + (`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above. +- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't + fail the run; **thresholds are SLOs expressed in code and they set the process exit + code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`. +- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's + view and the application's view sit on the same Grafana time axis. That side-by-side + is the single most convincing screenshot you can show Babak. +- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're + writing 60 lines of it. ### Resources -- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is - immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free. -- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks, - all free PDFs. Run the exercises solo with Ledgerline as the subject. -- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable - free, self-paced, deeper. -- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough - overview; pair it with your Postgres backup knowledge. +- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*, + *Thresholds*. Free, genuinely well written; ~2 h to work through. +- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free, + hands-on). +- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one. +- Grafana docs → *k6 output to Prometheus remote write*. ### Build -1. Write `docs/SLO.md`. For Ledgerline define, with justification: - - Availability SLI/SLO on `POST /api/transactions` - - Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not - from thin air) - - A freshness/lag SLI on the notifier consumer group - - A stated SLA that is deliberately looser than each SLO - - An RTO and RPO for the Postgres instance, with the backup cadence they imply - - An error budget policy: what happens at 50%, 75% and 100% consumed -2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**. - Your platform ships **Alertmanager** plus a Prometheus rules directory and the - sample rules `HostExporterDown` and `HostHighCPU`, so the grain is - *Prometheus rule files → Alertmanager*, templated by the Ansible role, with - `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but - be able to argue the alternative: Grafana unified alerting can evaluate across - datasources (including Loki queries) and lives with your dashboards, whereas - Alertmanager is metrics-only but is the ecosystem standard and gives you - grouping, deduplication, inhibition and silences for free. Note which you chose - and why in `docs/SLO.md`. -3. Add an error-budget panel to the overview dashboard: budget remaining, current - burn rate, projected exhaustion. -4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and - watch the fast-burn alert fire. +1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`, + sharing a `common.js` for setup and payload generation against random accounts. +2. Use **arrival-rate executors** for `average` and `stress`. Write down why. +3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`, + `http_req_failed rate<0.01`. +4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard. +5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result + sets. This before/after pair is your baseline for Phase 9. +6. Optional but impressive: have k6 send W3C trace context headers so load-test + requests appear as traces, letting you click from a k6 spike into a specific + trace. Check the current k6 docs for the tracing module's status — it has moved + between experimental and stable. -**Done when:** you can explain, in one paragraph each and without notes, the -difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule -decision. +**Done when:** `make load-stress` produces a Grafana view where you can see the +arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the +service that broke first. diff --git a/docs/roadmap/phase-8.md b/docs/roadmap/phase-8.md index fd5e85e..069bcc5 100644 --- a/docs/roadmap/phase-8.md +++ b/docs/roadmap/phase-8.md @@ -1,90 +1,91 @@ -# Phase 8 — Bottleneck hunting (the capstone) +# Phase 8 — SLI, SLO, SLA, error budgets, RTO, RPO *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)* **Status:** Not started -**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics, -dashboards, load-test baselines, and burn-rate alerts are the toolkit this -phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts. +**Message:** Depends on Phase 7's load-test baseline (the latency SLO should +come from measured numbers, not be picked from thin air). -**Time-box:** 2 evenings. This is where the whole thing pays off. +**Time-box:** 1–2 evenings. Mostly thinking, a little config. -### Concepts — how to actually answer "how do I find bottlenecks?" +### Concepts -Babak asked how you'd *think* about this, not for a list of tools. Here's the -method, roughly in order. The discipline is to go **top-down from the user, not -bottom-up from the infrastructure** — bottom-up is how people spend three hours -tuning a database that wasn't the problem. - -1. **Start at the SLI, not the server.** Which endpoint, which percentile, which - window, compared to what baseline? "It's slow" is not a starting point. "p95 of - `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is. -2. **Localise to a service** using RED + the service graph. Which node's duration - grew? Note that a slow *downstream* makes its *upstream* look slow too, so read - the graph from the leaves inward. -3. **Get to exemplar traces.** Click the latency spike through to a trace, or query - TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want - several slow traces plus one fast trace of the same endpoint as a control. -4. **Read the trace for five specific shapes:** - - **A long span** — the work itself is slow. Look at its attributes: which query, - which host, which parameters. - - **A gap** — wall-clock time not covered by any span. This is the interesting - one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a - blocked event loop, or a missing instrumentation library. Ledgerline's - `pool_starvation` and `blocking_event_loop` both show up as gaps. - - **Serialisation** — spans laid end to end that could have been concurrent. - Ledgerline's `no_batching` looks exactly like this. - - **Repetition** — the same span name 150 times in a row. That's your N+1. - - **Retries** — the same client span repeated 3–4 times with errors. That's - `retry_storm`, and note that it makes an *overload* worse, never better. -5. **Separate latency from saturation.** Switch from RED to USE on the suspected - resource: connection pool in-use vs pool size, stream consumer lag, CPU, - memory. A resource at 100% utilisation with a growing queue is saturated; - latency there is a *symptom*, not the cause. -6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency - equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10 - concurrent slots. If your pool is 5, you queue — and queueing latency grows - non-linearly as you approach saturation. This one formula explains most - connection-pool and worker-count incidents you'll ever see. -7. **Check for tail amplification** (Phase 6) before blaming any single service. -8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one - chaos flag, re-run the *identical* k6 profile, and compare the *same percentile* - over the *same duration*. One variable at a time. Never declare a fix from a - dashboard that "looks better." -9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest - becomes the bottleneck. Re-measure after every change; stop when you're inside - the SLO, not when you run out of ideas. -10. **Know what your tools can't see.** Traces won't show you CPU time inside a span, - or lock waits inside Postgres, or which line of Python burned the cycles. That's - where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous - profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over. +- **SLI** — a *measurement* of something users experience, expressed as a ratio of + good events to valid events. Not "CPU is at 60%". Rather: *the proportion of + `POST /api/transactions` requests that returned 2xx within 500 ms.* +- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction + requests succeed within 500 ms, measured over a rolling 28 days." +- **SLA** — an external, *contractual* promise, with consequences (refunds, credits). + Always **looser** than your SLO, so you have room to miss internally without + breaching commercially. If your SLA and SLO are the same number, you've made a + mistake. +- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1% + budget; over 30 days that's about **43 minutes**. The budget is the thing that + turns reliability from an argument into arithmetic: if there's budget left, ship + features and take risks; if it's spent, the policy says you stop feature work and + spend on reliability. That policy — agreed in advance by dev, ops and product — is + the whole point. +- **Burn rate.** How fast you're consuming budget relative to the window. + Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's + multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket) + is what stops you from either paging on every blip or noticing the slow bleed a + month late. +- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs, + and it's worth being precise because interviewers probe exactly here. + - **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long + from failure to restored service. It drives your architecture: replication, + warm standby, failover automation, runbook quality. + - **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured + in time: how far back can you afford to lose. It drives your **backup and + replication frequency**. RPO of 15 minutes means backups/WAL shipping at least + every 15 minutes. + - The relationship worth stating out loud: **SLO governs steady-state quality; RTO + and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says + "if Postgres dies we're back within 30 minutes." RPO says "and we lose at most + 5 minutes of transactions." Both feed the SLA. + - Both are **business decisions, not technical ones** — you derive them by asking + what an hour of downtime or an hour of lost data actually costs, then buying the + cheapest architecture that meets it. Tightening either is expensive and + non-linear. ### Resources -- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing - for saturation. -- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk). -- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best - thing ever written on why microservice latency behaves the way it does. -- Grafana docs → Tempo → **TraceQL**. You'll live in this. -- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know - these; connect them to the trace's DB spans. +- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is + immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free. +- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks, + all free PDFs. Run the exercises solo with Ledgerline as the subject. +- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable + free, self-paced, deeper. +- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough + overview; pair it with your Postgres backup knowledge. ### Build -1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at - `docs/PATHOLOGIES.md`.** -2. Run `make load-average` for 10 minutes. -3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven, - like a real incident doc: what you observed, what you hypothesised, what you - checked, what it turned out to be. Include screenshots. -4. When you think you've found them all, run `make chaos-clean`, enable them one at - a time, and confirm each signature. -5. *Then* read `docs/PATHOLOGIES.md` and score yourself. -6. Repeat once with a new random set. The second run is where the method actually - becomes yours. +1. Write `docs/SLO.md`. For Ledgerline define, with justification: + - Availability SLI/SLO on `POST /api/transactions` + - Latency SLI/SLO on the same (pick the threshold from your Phase 7 baseline, not + from thin air) + - A freshness/lag SLI on the notifier consumer group + - A stated SLA that is deliberately looser than each SLO + - An RTO and RPO for the Postgres instance, with the backup cadence they imply + - An error budget policy: what happens at 50%, 75% and 100% consumed +2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**. + Your platform ships **Alertmanager** plus a Prometheus rules directory and the + sample rules `HostExporterDown` and `HostHighCPU`, so the grain is + *Prometheus rule files → Alertmanager*, templated by the Ansible role, with + `alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but + be able to argue the alternative: Grafana unified alerting can evaluate across + datasources (including Loki queries) and lives with your dashboards, whereas + Alertmanager is metrics-only but is the ecosystem standard and gives you + grouping, deduplication, inhibition and silences for free. Note which you chose + and why in `docs/SLO.md`. +3. Add an error-budget panel to the overview dashboard: budget remaining, current + burn rate, projected exhaustion. +4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and + watch the fast-burn alert fire. -**Done when:** you find at least 4 of the enabled pathologies from telemetry alone, -and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague. +**Done when:** you can explain, in one paragraph each and without notes, the +difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule +decision. diff --git a/docs/roadmap/phase-9.md b/docs/roadmap/phase-9.md index c894b45..b9ad0b2 100644 --- a/docs/roadmap/phase-9.md +++ b/docs/roadmap/phase-9.md @@ -1,161 +1,90 @@ -# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry +# Phase 9 — Bottleneck hunting (the capstone) *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)* **Status:** Not started -**Message:** Not technically blocked by earlier phases (it stands on its own -as a judgment exercise), but sequenced last-but-one so Phase 8's incident to -re-diagnose in Kibana already exists. +**Message:** Depends on Phases 3–8 all being in place (traces, logs, metrics, +dashboards, load-test baselines, and burn-rate alerts are the toolkit this +phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts. -**Time-box:** 1–2 evenings. +**Time-box:** 2 evenings. This is where the whole thing pays off. -Babak's remaining questions are all *judgment* questions. He isn't testing whether -you can recite what Filebeat does; he's testing whether you have a framework for -choosing. Here's the framework, then the answers as worked examples. +### Concepts — how to actually answer "how do I find bottlenecks?" -### The framework — six axes to reason on +Babak asked how you'd *think* about this, not for a list of tools. Here's the +method, roughly in order. The discipline is to go **top-down from the user, not +bottom-up from the infrastructure** — bottom-up is how people spend three hours +tuning a database that wasn't the problem. -Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through: - -1. **Where does the data originate?** Inside your application (you can instrument it) - or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't - touch, syslog from a switch)? You can only use OTel SDKs where you own the code. -2. **What shape is it already in?** Structured JSON at the source, or unstructured - text needing grok/regex parsing? Parsing cost has to land *somewhere*. -3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate? -4. **What's the destination, and how many?** One backend forever, or several, or one - you expect to change? -5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent - on every host) distributes cost and adds resilience but multiplies config surface. - Parsing centrally concentrates cost and creates a single point of failure — but - one place to change rules. -6. **What already exists, and what does it cost?** Existing Elastic investment, - existing licences, team skills, and licensing constraints (Elastic's Filebeat and - Logstash are source-available under Elastic licensing; the OTel Collector, Fluent - Bit and Vector are permissively licensed). Migration is never free. - -### The answers - -**When to use Filebeat.** When you need a lightweight shipper that tails log *files* -from a host or container and forwards them with minimal transformation — and your -destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once -delivery with a registry of file offsets, backpressure handling, and prebuilt modules -for common services (nginx, MySQL, system logs) that come with parsing and dashboards -included. Reach for it when the source is a file you don't control the format of, the -app can't be modified, and Elasticsearch is where it's going. - -**When to use Logstash.** When you need real *transformation*: parsing unstructured -text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB), -splitting or merging events, conditional routing to multiple destinations, or -buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine -for events. It's heavyweight — a JVM — so you run a few of them centrally, not one -per host. - -**When to use Filebeat and Logstash together, and why.** This is the classic Elastic -architecture and the reason is a **division of labour between edge and centre**: - -- Filebeat runs on *every* host: cheap, tails files, handles offsets and - backpressure, ships onward. You want the thing running in a thousand places to be - small and boring. -- Logstash runs on a *few* central nodes: does the expensive grok parsing, - enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so - you want them in a handful of places you can redeploy, not on a thousand hosts. -- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's - persistent queue absorbs it and Filebeat backs off gracefully, rather than every - host queuing independently or dropping. - -So: Filebeat for *collection and delivery*, Logstash for *processing and routing*. -Use both when you have many hosts **and** non-trivial parsing/enrichment **and** -need central buffering. Skip Logstash when your logs are already structured JSON and -Elasticsearch ingest pipelines can do the little parsing that remains. - -**When to use OpenTelemetry.** - -- You own the application code and want telemetry generated *from inside it* — spans - with business context, custom metrics, logs already carrying trace IDs. -- You need **more than one signal**, correlated. This is the strongest argument: one - SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs - threaded through all three. Neither Filebeat nor Logstash does traces. -- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent - data model across all of them. -- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for - Jaeger by editing Collector YAML. -- You need distributed tracing at all. There is no serious vendor-neutral alternative. -- Your telemetry needs processing in flight — redaction, tail sampling, attribute - rewriting — without touching the app. - -**When *not* to use OpenTelemetry.** This is the question that separates people who -have used it from people who have read about it. - -- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary, - an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a - file-based collector. -- **You only need one signal from one source with no correlation.** A single nginx - access log going to a single log store: OTel is more machinery than the job needs. - Fluent Bit or Filebeat is simpler and finishes the job today. -- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature - Elastic estate with hundreds of Beats modules, dashboards and detection rules is - years of investment. "We should use OTel because it's the standard" is not a - business case. Note the direction of travel though: the Collector can already - receive from and export to most of these, so a *gradual* migration is usually the - right shape — often by putting a Collector or Vector in front of the existing - agents rather than ripping them out. -- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets, - where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes. -- **The signal is genuinely business analytics, not telemetry.** Clickstream data for - a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different - retention, different query patterns, different cost model. -- **Maturity gaps for a specific need in a specific language.** OTel logs support has - matured at different rates per language, and profiling is still alpha. Check the - status page for your language before betting on a signal. -- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks - on it. A two-person team that needs error tracking by Friday should install Sentry. - -The senior framing to give Babak: *these tools aren't competitors so much as -different layers.* OpenTelemetry is a **generation and transport standard** for -telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you -don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and -Elasticsearch are **stores**. Most real architectures use one from each layer, and -the interesting question is always which layer a given problem lives in. +1. **Start at the SLI, not the server.** Which endpoint, which percentile, which + window, compared to what baseline? "It's slow" is not a starting point. "p95 of + `POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is. +2. **Localise to a service** using RED + the service graph. Which node's duration + grew? Note that a slow *downstream* makes its *upstream* look slow too, so read + the graph from the leaves inward. +3. **Get to exemplar traces.** Click the latency spike through to a trace, or query + TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want + several slow traces plus one fast trace of the same endpoint as a control. +4. **Read the trace for five specific shapes:** + - **A long span** — the work itself is slow. Look at its attributes: which query, + which host, which parameters. + - **A gap** — wall-clock time not covered by any span. This is the interesting + one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a + blocked event loop, or a missing instrumentation library. Ledgerline's + `pool_starvation` and `blocking_event_loop` both show up as gaps. + - **Serialisation** — spans laid end to end that could have been concurrent. + Ledgerline's `no_batching` looks exactly like this. + - **Repetition** — the same span name 150 times in a row. That's your N+1. + - **Retries** — the same client span repeated 3–4 times with errors. That's + `retry_storm`, and note that it makes an *overload* worse, never better. +5. **Separate latency from saturation.** Switch from RED to USE on the suspected + resource: connection pool in-use vs pool size, stream consumer lag, CPU, + memory. A resource at 100% utilisation with a growing queue is saturated; + latency there is a *symptom*, not the cause. +6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency + equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10 + concurrent slots. If your pool is 5, you queue — and queueing latency grows + non-linearly as you approach saturation. This one formula explains most + connection-pool and worker-count incidents you'll ever see. +7. **Check for tail amplification** (Phase 7) before blaming any single service. +8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one + chaos flag, re-run the *identical* k6 profile, and compare the *same percentile* + over the *same duration*. One variable at a time. Never declare a fix from a + dashboard that "looks better." +9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest + becomes the bottleneck. Re-measure after every change; stop when you're inside + the SLO, not when you run out of ideas. +10. **Know what your tools can't see.** Traces won't show you CPU time inside a span, + or lock waits inside Postgres, or which line of Python burned the cycles. That's + where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous + profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over. ### Resources -- Better Stack — *Filebeat vs Logstash: How to Choose* (free article). -- SigNoz — *OpenTelemetry vs ELK Stack* (free article). -- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* — - read these to see how the Collector does the Filebeat/Logstash job itself. -- OpenTelemetry docs → *Instrumentation status by language* — for the maturity - caveats. +- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing + for saturation. +- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk). +- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best + thing ever written on why microservice latency behaves the way it does. +- Grafana docs → Tempo → **TraceQL**. You'll live in this. +- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know + these; connect them to the trace's DB spans. -### Build — and this is no longer hypothetical +### Build -Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml` -deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with -Filebeat reading Docker container logs and Logstash parsing JSON into daily -`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these -questions — the codebase he handed you contains both worlds. +1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at + `docs/PATHOLOGIES.md`.** +2. Run `make load-average` for 10 minutes. +3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven, + like a real incident doc: what you observed, what you hypothesised, what you + checked, what it turned out to be. Include screenshots. +4. When you think you've found them all, run `make chaos-clean`, enable them one at + a time, and confirm each signature. +5. *Then* read `docs/PATHOLOGIES.md` and score yourself. +6. Repeat once with a new random set. The second run is where the method actually + becomes yours. -1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs** - flow down two pipelines: Filebeat → Logstash → Elasticsearch, and - Alloy → Loki. Compare them on the same data. -2. Test the direct ingest path the README documents — pipe a JSON line into - Logstash's TCP input on 5000, then query Elasticsearch — so you understand where - parsing happens before you change it. -3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter - that parses them. Then do the same job with Alloy's `loki.process` stages. Time - both. Note which one you'd rather debug at 2 a.m. -4. Compare the *investigation experience* directly: take one incident you already - diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that - you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana - wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL - for some questions, and Loki's cost model genuinely beats Elasticsearch's for - others. -5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic - licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are - permissively licensed. For some employers that alone decides it. -6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most - valuably — what you actually found annoying about each. Having run both stacks - against the same application is the difference between reciting a comparison and - arguing one. +**Done when:** you find at least 4 of the enabled pathologies from telemetry alone, +and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague. diff --git a/docs/roadmap/summary.md b/docs/roadmap/summary.md index d115381..403c9c5 100644 --- a/docs/roadmap/summary.md +++ b/docs/roadmap/summary.md @@ -6,7 +6,7 @@ for the phase you're actually working on. This file exists so an agent (or you) can check overall status and sequencing without loading every phase's full detail.* -Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase +Twelve phases (0 through 11 — Phase 0 has two halves, 0a and 0b). Each phase file follows the same shape: **Status**, **Message** (what's done, notes, warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. @@ -16,56 +16,60 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. |---|---|---|---|---|---| | 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` | | 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations | -| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo | -| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | -| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation | -| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars | -| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` | -| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles | -| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts | -| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` | -| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` | -| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo | +| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` | +| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo | +| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | +| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation | +| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars | +| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` | +| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles | +| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts | +| 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` | +| 10 | [`phase-10.md`](phase-10.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` | +| 11 | [`phase-11.md`](phase-11.md) | Not started | Delivery | 1 evening | README + rehearsed demo | ## What each phase is, in one line - **0 — Build the base project, test-first.** Write the black-box test suite, then implement the seven-service system against it as a frozen contract. **Done** — see `phase-0.md` for the full implementation report. -- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector +- **1 — Deployment.** Put Ledgerline on a server with Docker Compose behind + Traefik and a Let's Encrypt certificate, served at a real domain. Plumbing + only — no telemetry, no app changes. **Done** — see `phase-1.md`. +- **2 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via Ansible and get one auto-instrumented trace into Tempo. -- **2 — Distributed tracing.** The heart of the task: auto- and +- **3 — Distributed tracing.** The heart of the task: auto- and manual-instrument all seven services, and hand-propagate trace context across the Redis Stream hop, which carries none by default. -- **3 — Structured logs, correlated to traces.** Replace prose logging with +- **4 — Structured logs, correlated to traces.** Replace prose logging with structured events carrying a lowercase `trace_id`, so Loki and Tempo link in both directions. -- **4 — Metrics.** Instrument RED metrics per service, understand cardinality +- **5 — Metrics.** Instrument RED metrics per service, understand cardinality limits, and route app metrics to Prometheus given Alloy drops OTLP metrics silently. Tempo's metrics-generator already produces span metrics and the service graph — don't duplicate it in the collector. -- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the +- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the Grafana Foundation SDK and ship it through the Ansible role, so "destroy and recreate in seconds" is a real, repeatable command. -- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't +- **7 — Load testing with k6, and percentiles.** Learn why percentiles can't be averaged, what coordinated omission is, and drive Ledgerline with arrival-rate executors to get a clean before/after load profile. -- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and - SLOs from the Phase 6 baseline (not from thin air), write an error-budget +- **8 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and + SLOs from the Phase 7 baseline (not from thin air), write an error-budget policy, and wire multi-burn-rate alerts. -- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set +- **9 — Bottleneck hunting (the capstone).** Given a random, undisclosed set of chaos flags, find the pathologies from telemetry alone using a top-down, hypothesis-driven method, and write it up as an incident doc. -- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs +- **10 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs OpenTelemetry on six concrete axes, using the platform's real second stack (Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint. -- **10 — Package the delivery.** README, a rehearsed five-minute demo, the +- **11 — Package the delivery.** README, a rehearsed five-minute demo, the written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`, `CONCEPTS.md`), and an honest "what I'd do differently in production" section. ## If you have to cut scope -Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not -cut Phase 8** — it's the only phase that proves you learned anything. +Cut Phase 10's build (keep the writing) and Phase 7's soak test first. **Do not +cut Phase 9** — it's the only phase that proves you learned anything.