mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add deployment phase and Traefik compose override

Browse Source 
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked
Done) between the base project and the OTel concepts phase, and shift
the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md
references (spoiler rules now target Phase 9, the bottleneck hunt).

Add docker-compose.traefik.yml: an override that fronts the gateway with
Traefik and Let's Encrypt and removes all other published host ports.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-09 15:38:29 +03:30
co-authored by Claude Sonnet 5.5
parent b2112e55e1
commit 5e514ecf25
17 changed files with 1080 additions and 937 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/roadmap/intro.md
+39 -38
View File
@@ -1,7 +1,7 @@
# Observability & APM — Delivery Roadmap for Babak's Task
*Part of `docs/roadmap/` — platform background and reading guidance shared by
every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`;
every phase. Per-phase detail lives in `phase-0.md` through `phase-11.md`;
[`summary.md`](summary.md) gives the one-page overview and status of all of
them.*
@@ -90,16 +90,16 @@ learn OTel first, every backend becomes "point OTLP at a different URL."
One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
early, in parallel with OTel metrics — because it *constrains what you're allowed to
emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
(Phases 4–5), not up front.
(Phases 5–6), not up front.
Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
part of the Grafana stack and you do not need them for this task. Phase 9 covers
part of the Grafana stack and you do not need them for this task. Phase 10 covers
them as a judgment question, which is exactly how Babak framed them.
### Verdict on the OneUptime article you found
**Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
teaching resource. Read it in **Phase 4/5**, not now; it assumes you already
teaching resource. Read it in **Phase 5/6**, not now; it assumes you already
understand spans, span metrics and PromQL.
What's genuinely good in it, and rare to find in one place:
@@ -133,7 +133,7 @@ Two caveats:
This changes *where* your telemetry goes and *how the pipeline is configured*. It
does not change the learning arc, the app, or the test suite. Read this before
Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9.
Phase 2; it rewrites parts of Phases 2, 4, 5, 6, 8 and 10.
### What the repo is
@@ -199,7 +199,7 @@ Prometheus`. The shipped config does not do that:
Neither is wrong; the diagram shows the intended architecture and the code
implements a simpler one. Closing that gap — adding `prometheus.scrape` +
`prometheus.remote_write` to Alloy so metrics really do flow through the collector —
is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth
is a genuinely good Phase 5 exercise, and **noticing the discrepancy at all is worth
mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
exactly the skill this task is meant to build.
@@ -264,7 +264,7 @@ Tempo and remote-written into Prometheus (which runs with
connectors to Alloy** — you'd be generating the same series twice under different
names. The metric names differ from the OneUptime article's; go find the actual
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.)
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.)
**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
datasource is provisioned with:
@@ -295,16 +295,17 @@ your deliverable, and the natural home for it is *inside this Ansible role*.
| Phase | Impact |
|---|---|
| 0 / 0a | **None.** App and test suite are unaffected. |
| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
| 2 | Endpoint changes to Alloy; tracing work itself is identical. |
| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
| 6 | None. |
| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
| 8 | None, plus TraceQL metrics from `local-blocks`. |
| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
| 10 | Demo now includes `ansible-playbook` rebuilding the platform. |
| 1 | **New.** Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes. |
| 2 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
| 3 | Endpoint changes to Alloy; tracing work itself is identical. |
| 4 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
| 5 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
| 6 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
| 7 | None. |
| 8 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
| 9 | None, plus TraceQL metrics from `local-blocks`. |
| 10 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
| 11 | Demo now includes `ansible-playbook` rebuilding the platform. |
### Where to run it
@@ -326,7 +327,7 @@ Is this repo a **platform you must consume as-is**, or a **starting point you ma
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
extend `group_vars`. If Babak intends it as a fixed platform, several
recommendations below change — ask before Phase 3.
recommendations below change — ask before Phase 4.
---
@@ -340,8 +341,8 @@ subtitle-to-notes workflow.
| Resource | Format | Length | Cost | Best for |
|---|---|---|---|---|
| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. |
| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. |
| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 4–6. Comes with browser-based Killercoda playgrounds. |
| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 3–5. |
If budget allows, buy both — they overlap by maybe 30% and the second half of each
is different. If only one: take the **OpenTelemetry** one, because instrumentation
@@ -351,27 +352,27 @@ is the part you have to *write*, and Grafana's own docs are excellent for the re
| Resource | Format | Length | Use in |
|---|---|---|---|
| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. |
| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. |
| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 2. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 3. The best Python-specific overview, with a companion repo. |
| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 3. Auto- vs manual-instrumentation side by side. |
| **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. |
| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 |
| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. |
| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on |
| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 |
| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. |
| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 |
| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. |
| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 |
| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 |
| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 2. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 2–5. The neutral source of truth. |
| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 6–7 |
| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 2–5. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 5, 9. This is where your RED metrics and service graph actually come from. |
| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 10, now hands-on |
| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 8 |
| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 8. Self-run the exercises with Ledgerline as the subject service. |
| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 8, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 9 |
| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 5. The clearest explanation anywhere of what cardinality actually costs. |
| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 3 |
| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 10 |
**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on
**Where to run it:** Docker Compose on your laptop for Phases 0–9. Don't put this on
your k8s cluster until everything works locally — debugging instrumentation and
debugging Kubernetes at the same time is how these projects die. Porting it to the
cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half
cluster is a good Phase 11+ follow-up, and it's the natural bridge to the DevOps half
Babak has planned for you.
docs/roadmap/phase-0.md
+3 -3
View File
@@ -67,7 +67,7 @@ deliberate brokenness.
### Why TDD here, when the app itself doesn't matter
The microservices are throwaway. The *properties* aren't. Everything from Phase 2
The microservices are throwaway. The *properties* aren't. Everything from Phase 3
onward assumes the system still behaves a specific way: seven real services in
the path, async work genuinely off the request path, five hot accounts owning a
third of the rows, and thirteen pathologies that actually degrade something. You
@@ -114,13 +114,13 @@ Done when `make test` is green three consecutive runs and
### Then, yourself
1. **Read every service's `app.py` end to end.** You need to know this codebase
cold, because from Phase 2 onward you're modifying all seven services.
cold, because from Phase 3 onward you're modifying all seven services.
2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
then try to figure out what's wrong using only `docker compose logs`.** Give it
twenty real minutes. You will fail, and the specific way you fail — no timing,
no causality, no cross-service correlation, a `X-Request-Id` that dies at the
Redis Stream — is the motivation for everything that follows. Write down what
you wished you had. That list is your acceptance criteria for Phase 8.
you wished you had. That list is your acceptance criteria for Phase 9.
**Done when:** `make test` is green three runs in a row, all nine acceptance
criteria in the base spec pass, and you have that written list of frustrations.
docs/roadmap/phase-1.md
+64 -79
View File
@@ -1,89 +1,74 @@
# Phase 1 — The model, and one signal end to end
# Phase 1 — Deployment: Ledgerline on a server behind Traefik
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
**Status:** Not started
**Status:** Done
**Message:** Blocked on nothing — Phase 0 is done and the platform is
described above. Waiting to begin.
**Message:** Ledgerline is running on a remote server and served over HTTPS at
**https://ledgerline.vione.cloud**. Verified against the live deployment, not
just the config: the frontend loads, `/api/health/all` reports every service
`ok`, and the origin serves a real Let's Encrypt certificate (valid until
2027-01-07, auto-renewed by Traefik).
**Time-box:** one evening.
- **Target:** the throwaway server `czy` (Ubuntu 24.04, 1 vCPU, 1 GB RAM), repo
at `/opt/apps/LedgerLine`. Docker 29.9 and Compose v5.6 installed from
Docker's apt repo. A 2 GB swapfile was added (persisted in `/etc/fstab`) —
nine containers do not fit in 1 GB of RAM without it, and steady state still
uses roughly 700 MB plus swap.
- **How it's wired:** `docker-compose.traefik.yml` is an *override*, not an
edit of `docker-compose.yml`, so local development is unchanged. It adds
Traefik v3.6 on ports 80/443 (Docker provider, `exposedByDefault=false`,
HTTP→HTTPS redirect, Let's Encrypt via HTTP-01), routes
``Host(`ledgerline.vione.cloud`)`` to the gateway on port 8000, and strips
the published host ports from every other service (`ports: !reset []`) so
Postgres, Redis and the services are reachable only on the Compose network.
- **No application or `tests/` changes.** The Dockerfile and base compose file
are untouched; schema applied and data seeded on the server with the same
steps as `make seed` (800 accounts, 250,000 transactions).
- **Deploy / redeploy command** (always both files — a bare `docker compose up`
would use the base file alone and publish every port):
**Why:** you need the vocabulary, but more importantly you need to see one trace
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
than abstract.
```bash
docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --build
```
- **Cloudflare gotchas hit on the way:**
- The first certificate attempt failed with an ACME 404 because the
Cloudflare DNS record did not yet point at the server — a packet capture
on the server showed no inbound traffic from Cloudflare at all. Traefik
only retries issuance on restart, so after fixing the record it needed
`docker compose ... restart traefik`.
- With the orange-cloud proxy on, the SSL/TLS mode must be **Full** —
**Flexible** combined with Traefik's HTTPS redirect causes a redirect loop.
- **Caveats to remember:** the deployment is public, including the
`/api/admin/chaos` endpoints (anyone who finds them can flip flags);
internet scanners probed the site within seconds of it going live. The
server is disposable by design, and Traefik's state lives in the
`letsencrypt` volume, which is lost on an OS reinstall (a new certificate is
simply issued on the next run).
**Time-box:** 1 evening.
**Deliverable:** the unmodified-behaviour Ledgerline system reachable at a real
domain over HTTPS, reproducible with one compose command.
### Concepts
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
(known unknowns): dashboards and alerts built from predefined aggregates.
Observability is a property of the system: can you ask a *new* question about
production, that you never anticipated, without shipping new code? Charity Majors'
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
practical implication for you: emit **wide, high-dimensional events** (rich span
attributes), not just pre-aggregated counters.
- **The OTel architecture split: API vs SDK vs Collector.**
- The **API** is what your code and third-party instrumentation libraries call.
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
library can be instrumented without forcing a dependency on any implementation.
- The **SDK** is the engine: sampling, batching, context propagation, export.
Application owners configure the SDK; library authors only touch the API.
- The **Collector** is a standalone process that receives, processes and exports
telemetry. Optional, but you should use one.
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
`service.version`, `deployment.environment`) and is attached to every signal from
that process. Attributes describe the individual span/metric/log.
- **Semantic conventions**: standardised attribute names (`http.request.method`,
`db.system`, `server.address`). Use them for anything standard; namespace your own
(`ledgerline.account.id`). This is what makes cross-service, cross-language
correlation work — and it's what Grafana's prebuilt dashboards key off.
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
aware of, not learning yet.)
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
seven services. The Collector also does the things you don't want in your app:
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
destinations.
### Resources
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
M.E.L.T. and context-propagation sections; skip the Node code.
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
it — just see what "done" looks like.
### Build
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
`ansible-galaxy collection install -r requirements.yml`, change
`grafana_admin_password` (vault it), then
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
Then open the Alloy UI and **read the component graph** — it shows you the exact
pipeline, including the fact that the OTLP receiver has only a traces output.
Seeing that in the UI before you get bitten by it is worth ten minutes.
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
a single container with everything pre-wired. Useful for a first trace, but
don't build on it — its pipeline differs from your real platform in exactly the
ways that matter.)*
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
auto-instrumentation:
```
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```
and change its command to
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
with `OTEL_SERVICE_NAME=sanctions`,
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
Tempo directly.
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
**Done when:** you can see a single-service trace in Tempo, and you can explain out
loud what the Resource attributes on it mean and where they came from.
- **Traefik's model:** *entrypoints* (the ports it listens on), *routers*
(rule → service), *services* (where to send traffic), *middlewares*, and
*providers* (where config comes from — here, Docker labels on containers).
Compare with nginx, where the same information lives in one static config
file.
- **Compose override files.** `-f base -f override` merges the files;
`!reset` clears an inherited key (here, `ports`) instead of appending to it.
- **ACME HTTP-01 behind a CDN.** Let's Encrypt must be able to reach port 80 on
the *hostname's actual origin*. With Cloudflare proxying, that only works
once the DNS record points at the right server.
- **Why this phase exists before observability.** The telemetry work in
Phases 2–11 is more convincing when the system under observation is a real
deployment rather than a laptop, and the platform (`observability-ansible`)
is already a remote deployment — Ledgerline now sits on the same footing.
**Done when:** `https://ledgerline.vione.cloud` shows the frontend with a valid
certificate; `/api/health/all` is all `ok`; only ports 22, 80 and 443 are
reachable on the server; and the deploy is one documented compose command.
docs/roadmap/phase-10.md
+153 -32
View File
@@ -1,40 +1,161 @@
# Phase 10 — Package the delivery
# Phase 10 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)*
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`phase-11.md`](phase-11.md)*
**Status:** Not started
**Message:** Depends on every prior phase being done — this is the wrap-up,
not independent work.
**Message:** Not technically blocked by earlier phases (it stands on its own
as a judgment exercise), but sequenced last-but-one so Phase 9's incident to
re-diagnose in Kibana already exists.
**Time-box:** one evening.
**Time-box:** 1–2 evenings.
Babak gave you a task; deliver it like an engineer, not like a student.
Babak's remaining questions are all *judgment* questions. He isn't testing whether
you can recite what Filebeat does; he's testing whether you have a framework for
choosing. Here's the framework, then the answers as worked examples.
1. **`README.md`** — architecture diagram, one-command start, what each container is
for, and a "here's the demo" script.
2. **The five-minute demo path**, rehearsed:
- Delete every dashboard in the Grafana UI, then `ansible-playbook
playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
from code, one command.
- Submit one transaction from the frontend, then follow it through Grafana: RED
dashboard → latency spike → exemplar → trace → span → logs for that span → back
to the trace. One request, all three signals, no searching.
- `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
and the service map turn red at a specific node.
- Show `docs/INVESTIGATION.md` and how you found the pathologies.
3. **The written answers** Babak asked for, as documents in the repo, not as a
conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
`docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
RTO/RPO/SLA/SLO/error budget in your own words.
4. **The test suite as part of the story.** Show `make test` green, then show the
three generations of the call-counting assertion side by side —
`docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
each signal bought you. Most people demo dashboards; almost nobody can show
their observability work paying off inside their own test suite.
5. **A "what I'd do differently in production" section.** Tail sampling instead of
100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
you *didn't* do, and why, is the strongest signal of seniority in the whole
delivery.
### The framework — six axes to reason on
Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
1. **Where does the data originate?** Inside your application (you can instrument it)
or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
2. **What shape is it already in?** Structured JSON at the source, or unstructured
text needing grok/regex parsing? Parsing cost has to land *somewhere*.
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
4. **What's the destination, and how many?** One backend forever, or several, or one
you expect to change?
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
on every host) distributes cost and adds resilience but multiplies config surface.
Parsing centrally concentrates cost and creates a single point of failure — but
one place to change rules.
6. **What already exists, and what does it cost?** Existing Elastic investment,
existing licences, team skills, and licensing constraints (Elastic's Filebeat and
Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
Bit and Vector are permissively licensed). Migration is never free.
### The answers
**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
from a host or container and forwards them with minimal transformation — and your
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
included. Reach for it when the source is a file you don't control the format of, the
app can't be modified, and Elasticsearch is where it's going.
**When to use Logstash.** When you need real *transformation*: parsing unstructured
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
splitting or merging events, conditional routing to multiple destinations, or
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
per host.
**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
architecture and the reason is a **division of labour between edge and centre**:
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
backpressure, ships onward. You want the thing running in a thousand places to be
small and boring.
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
you want them in a handful of places you can redeploy, not on a thousand hosts.
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
persistent queue absorbs it and Filebeat backs off gracefully, rather than every
host queuing independently or dropping.
So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
need central buffering. Skip Logstash when your logs are already structured JSON and
Elasticsearch ingest pipelines can do the little parsing that remains.
**When to use OpenTelemetry.**
- You own the application code and want telemetry generated *from inside it* — spans
with business context, custom metrics, logs already carrying trace IDs.
- You need **more than one signal**, correlated. This is the strongest argument: one
SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
threaded through all three. Neither Filebeat nor Logstash does traces.
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
data model across all of them.
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
Jaeger by editing Collector YAML.
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
rewriting — without touching the app.
**When *not* to use OpenTelemetry.** This is the question that separates people who
have used it from people who have read about it.
- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
file-based collector.
- **You only need one signal from one source with no correlation.** A single nginx
access log going to a single log store: OTel is more machinery than the job needs.
Fluent Bit or Filebeat is simpler and finishes the job today.
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
Elastic estate with hundreds of Beats modules, dashboards and detection rules is
years of investment. "We should use OTel because it's the standard" is not a
business case. Note the direction of travel though: the Collector can already
receive from and export to most of these, so a *gradual* migration is usually the
right shape — often by putting a Collector or Vector in front of the existing
agents rather than ripping them out.
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
retention, different query patterns, different cost model.
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
matured at different rates per language, and profiling is still alpha. Check the
status page for your language before betting on a signal.
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
on it. A two-person team that needs error tracking by Friday should install Sentry.
The senior framing to give Babak: *these tools aren't competitors so much as
different layers.* OpenTelemetry is a **generation and transport standard** for
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
Elasticsearch are **stores**. Most real architectures use one from each layer, and
the interesting question is always which layer a given problem lives in.
### Resources
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
read these to see how the Collector does the Filebeat/Logstash job itself.
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
caveats.
### Build — and this is no longer hypothetical
Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
Filebeat reading Docker container logs and Logstash parsing JSON into daily
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
questions — the codebase he handed you contains both worlds.
1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
Alloy → Loki. Compare them on the same data.
2. Test the direct ingest path the README documents — pipe a JSON line into
Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
parsing happens before you change it.
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
that parses them. Then do the same job with Alloy's `loki.process` stages. Time
both. Note which one you'd rather debug at 2 a.m.
4. Compare the *investigation experience* directly: take one incident you already
diagnosed in Phase 9 and re-run it in Kibana Discover. What can you do there that
you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
for some questions, and Loki's cost model genuinely beats Elasticsearch's for
others.
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
permissively licensed. For some employers that alone decides it.
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
valuably — what you actually found annoying about each. Having run both stacks
against the same application is the difference between reciting a comparison and
arguing one.
docs/roadmap/phase-11.md
+40
View File
@@ -0,0 +1,40 @@
# Phase 11 — Package the delivery
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-10.md`](phase-10.md) · Next: [`summary.md`](summary.md)*
**Status:** Not started
**Message:** Depends on every prior phase being done — this is the wrap-up,
not independent work.
**Time-box:** one evening.
Babak gave you a task; deliver it like an engineer, not like a student.
1. **`README.md`** — architecture diagram, one-command start, what each container is
for, and a "here's the demo" script.
2. **The five-minute demo path**, rehearsed:
- Delete every dashboard in the Grafana UI, then `ansible-playbook
playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
from code, one command.
- Submit one transaction from the frontend, then follow it through Grafana: RED
dashboard → latency spike → exemplar → trace → span → logs for that span → back
to the trace. One request, all three signals, no searching.
- `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
and the service map turn red at a specific node.
- Show `docs/INVESTIGATION.md` and how you found the pathologies.
3. **The written answers** Babak asked for, as documents in the repo, not as a
conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
`docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
RTO/RPO/SLA/SLO/error budget in your own words.
4. **The test suite as part of the story.** Show `make test` green, then show the
three generations of the call-counting assertion side by side —
`docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
each signal bought you. Most people demo dashboards; almost nobody can show
their observability work paying off inside their own test suite.
5. **A "what I'd do differently in production" section.** Tail sampling instead of
100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
you *didn't* do, and why, is the strongest signal of seniority in the whole
delivery.
docs/roadmap/phase-2.md
+71 -98
View File
@@ -1,116 +1,89 @@
# Phase 2 — Distributed tracing
# Phase 2 — The model, and one signal end to end
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
**Status:** Not started
**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
described above. Waiting to begin.
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
**Time-box:** one evening.
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
(Resource, providers, exporters, Collector wiring) is shared across all three
signals — do it once here and logs and metrics become incremental. Second, tracing
is what Babak actually asked for ("trace a single request, see where it stuck"),
and once trace IDs exist, log correlation in Phase 3 works on the first try instead
of needing rework.
**Why:** you need the vocabulary, but more importantly you need to see one trace
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
than abstract.
### Concepts
- **Trace / span / span context.** A trace is one request's journey; a span is one
timed operation within it. Each span carries a trace ID, its own span ID, and its
parent's span ID. That parent pointer is the entire data structure.
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
these right is what makes the service graph render correctly.
- **Context propagation.**
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
carry the context explicitly.
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
- The classic bug: any boundary that doesn't carry those headers **starts a brand
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
producer and `.extract(...)` on the consumer, then starting the consumer span
with that context as parent. This is the single most valuable exercise in the
phase — most people never do it manually and never really understand it.
- Related: `links` (for a consumer span that batches messages from many traces)
and **Baggage** (arbitrary key/values propagated alongside trace context —
useful, and a foot-gun, because it's sent on every hop and is visible to
downstream systems; never put secrets in it).
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
that show *which service* was slow but not *why*.
- **Span attributes vs metric labels.** Spans are the right home for
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
with volume, not with distinct values. This is why "put it on the span, not the
metric" is the standing rule.
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
That's how the exception ends up attached to the trace and how the `spanmetrics`
connector in Phase 4 will count your error rate.
- **Sampling — the concept now, the config later.**
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
errors at the same rate as successes.
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
traces, but must run in the Collector and buffer spans in memory.
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
now; you'll configure tail sampling in the Collector in Phase 5.
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
unilaterally drop half of a trace its caller decided to keep.
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
(known unknowns): dashboards and alerts built from predefined aggregates.
Observability is a property of the system: can you ask a *new* question about
production, that you never anticipated, without shipping new code? Charity Majors'
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
practical implication for you: emit **wide, high-dimensional events** (rich span
attributes), not just pre-aggregated counters.
- **The OTel architecture split: API vs SDK vs Collector.**
- The **API** is what your code and third-party instrumentation libraries call.
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
library can be instrumented without forcing a dependency on any implementation.
- The **SDK** is the engine: sampling, batching, context propagation, export.
Application owners configure the SDK; library authors only touch the API.
- The **Collector** is a standalone process that receives, processes and exports
telemetry. Optional, but you should use one.
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
`service.version`, `deployment.environment`) and is attached to every signal from
that process. Attributes describe the individual span/metric/log.
- **Semantic conventions**: standardised attribute names (`http.request.method`,
`db.system`, `server.address`). Use them for anything standard; namespace your own
(`ledgerline.account.id`). This is what makes cross-service, cross-language
correlation work — and it's what Grafana's prebuilt dashboards key off.
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
aware of, not learning yet.)
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
seven services. The Collector also does the things you don't want in your app:
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
destinations.
### Resources
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
practical FastAPI-specific setup, including batch/queue env vars.
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
reading.
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
side by side. Look specifically at how they mix auto-instrumentation with
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
This is the "professional implementation" you asked to study.
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
M.E.L.T. and context-propagation sections; skip the Node code.
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
it — just see what "done" looks like.
### Build
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
service calls it once at startup. Keep it under 60 lines.
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
`ledgerline.tx.amount_minor`).
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
inject trace context into the message fields. Consumers (`notifier`, `audit`):
extract, and start a `CONSUMER` span with the extracted context as parent.
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
more.
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
the "before" state so you could measure against it, and from here it's
actively wrong. Everything else in `tests/` must still pass: instrumenting
the system is not allowed to change its behaviour, and if `make test` goes red
after adding spans, you've broken something real. That is exactly the safety
net you built it for.
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
`ansible-galaxy collection install -r requirements.yml`, change
`grafana_admin_password` (vault it), then
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
Then open the Alloy UI and **read the component graph** — it shows you the exact
pipeline, including the fact that the OTLP receiver has only a traces output.
Seeing that in the UI before you get bitten by it is worth ten minutes.
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
a single container with everything pre-wired. Useful for a first trace, but
don't build on it — its pipeline differs from your real platform in exactly the
ways that matter.)*
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
auto-instrumentation:
```
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```
and change its command to
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
with `OTEL_SERVICE_NAME=sanctions`,
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
Tempo directly.
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
### Done when
- One request produces **one connected trace** covering all seven services,
including both async consumers.
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
ate the time.
- You can explain why `blocking_event_loop` looks different in a trace than
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
before spans start, plus every concurrent request in that process degrading
together.)
**Done when:** you can see a single-service trace in Tempo, and you can explain out
loud what the Resource attributes on it mean and where they came from.
docs/roadmap/phase-3.md
+99 -82
View File
@@ -1,99 +1,116 @@
# Phase 3 — Structured logs, correlated to traces
# Phase 3 — Distributed tracing
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
**Status:** Not started
**Message:** Depends on Phase 2 (trace IDs must exist before log correlation
means anything).
**Message:** Depends on Phase 2's `common/telemetry.py` bootstrap existing first.
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
(Resource, providers, exporters, Collector wiring) is shared across all three
signals — do it once here and logs and metrics become incremental. Second, tracing
is what Babak actually asked for ("trace a single request, see where it stuck"),
and once trace IDs exist, log correlation in Phase 4 works on the first try instead
of needing rework.
### Concepts
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
events: a stable event name plus typed fields (`event="transaction_settled"`,
`account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
Consistent field names across all seven services — `account_id` everywhere, never
`accountId` in one place and `acct` in another. This is what makes LogQL queries
possible at all.
- **Trace–log correlation.** Every log line emitted inside an active span must carry
`trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
"one pane of glass": click a span → see its logs; click a log line → open its
trace. Both directions are configured, and you'll set both up.
- **Two ways to ship logs, and which one your platform actually does:**
- **stdout → collector scraping → Loki.** The container-native way. Your app just
prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
standard in Kubernetes. **This is the only path wired in your Alloy config**
(`loki.source.docker` reading the Docker socket).
- **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
with the same Resource attributes attached automatically. Cleaner correlation,
but the app owns delivery. **On your platform this currently goes nowhere** —
Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
dropped in silence.
- Do both anyway: use stdout for most services, and *extend Alloy* to wire the
OTLP logs path for one or two. Then argue the trade-off from experience. That
comparison — including the fact that a misconfigured pipeline fails silently
rather than loudly — is precisely what Babak will probe.
- **The field name is not a style choice here.** Your Loki datasource is provisioned
with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
A log line must carry lowercase `trace_id`. OpenTelemetry's logging
auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
logs and traces both arriving, no errors anywhere, and no clickable link. Hence
structlog with an explicit processor rather than the auto-injection shortcut.
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
your entire fleet, and Loki bills/indexes on volume.
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
`json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
redaction into the processor chain, not into each call site.
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
not log content. Labels should be few and low-cardinality (`service`, `env`,
`level`). Everything else lives in the log line body and is filtered at query time.
Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
in the JSON body and use a **derived field** to make it clickable.
- **Trace / span / span context.** A trace is one request's journey; a span is one
timed operation within it. Each span carries a trace ID, its own span ID, and its
parent's span ID. That parent pointer is the entire data structure.
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
these right is what makes the service graph render correctly.
- **Context propagation.**
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
carry the context explicitly.
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
- The classic bug: any boundary that doesn't carry those headers **starts a brand
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
producer and `.extract(...)` on the consumer, then starting the consumer span
with that context as parent. This is the single most valuable exercise in the
phase — most people never do it manually and never really understand it.
- Related: `links` (for a consumer span that batches messages from many traces)
and **Baggage** (arbitrary key/values propagated alongside trace context —
useful, and a foot-gun, because it's sent on every hop and is visible to
downstream systems; never put secrets in it).
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
that show *which service* was slow but not *why*.
- **Span attributes vs metric labels.** Spans are the right home for
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
with volume, not with distinct values. This is why "put it on the span, not the
metric" is the standing rule.
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
That's how the exception ends up attached to the trace and how the `spanmetrics`
connector in Phase 5 will count your error rate.
- **Sampling — the concept now, the config later.**
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
errors at the same rate as successes.
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
traces, but must run in the Collector and buffer spans in memory.
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
now; you'll configure tail sampling in the Collector in Phase 6.
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
unilaterally drop half of a trace its caller decided to keep.
### Resources
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
setup, including the hex formatting for trace/span IDs.
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
`OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
fields, if you'd rather not hand-roll a processor.
- Udemy backbone: the Loki + LogQL sections.
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
practical FastAPI-specific setup, including batch/queue env vars.
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
reading.
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
side by side. Look specifically at how they mix auto-instrumentation with
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
This is the "professional implementation" you asked to study.
### Build
1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
processor that pulls the current span context and injects `trace_id` / `span_id`.
2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
handler: request received (or rely on the span), business decision, error.
3. Add a redaction processor with a denylist of field names.
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
with a `container` label (Alloy's relabel rule sets it from the Docker container
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
the receiver shows traffic, and the pipeline shows no logs output. Now extend
`config.alloy.j2` — add a `logs` output on the receiver plus an
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
That is the most transferable half-hour in the whole project.
6. Confirm the provisioned correlation works in both directions: Loki's
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
are verifying, not writing. If a link doesn't appear, the regex didn't match
your field name — check that before anything else.
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
(`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
in `docker compose logs` — the only cross-service call counter that existed at
Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
with LogQL over structured fields instead. This is a small change and a
pointed one: notice how much shorter and more precise the assertion gets, and
that it now works without shelling out to Docker at all. That delta is the
value of structured logging, measured on your own code.
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
service calls it once at startup. Keep it under 60 lines.
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
`ledgerline.tx.amount_minor`).
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
inject trace context into the message fields. Consumers (`notifier`, `audit`):
extract, and start a `CONSUMER` span with the extracted context as parent.
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
more.
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
the "before" state so you could measure against it, and from here it's
actively wrong. Everything else in `tests/` must still pass: instrumenting
the system is not allowed to change its behaviour, and if `make test` goes red
after adding spans, you've broken something real. That is exactly the safety
net you built it for.
**Done when:** from a slow trace you can click into that request's logs, and from
any log line you can click into its trace — in both directions, in two clicks.
### Done when
- One request produces **one connected trace** covering all seven services,
including both async consumers.
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
ate the time.
- You can explain why `blocking_event_loop` looks different in a trace than
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
before spans start, plus every concurrent request in that process degrading
together.)
docs/roadmap/phase-4.md
+82 -109
View File
@@ -1,126 +1,99 @@
# Phase 4 — Metrics
# Phase 4 — Structured logs, correlated to traces
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
**Status:** Not started
**Message:** Not blocked technically (metrics don't strictly need tracing
first), but sequenced after Phase 3 per the roadmap's ordering.
**Message:** Depends on Phase 3 (trace IDs must exist before log correlation
means anything).
**Time-box:** 2 evenings.
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
### Concepts
- **Instrument types.**
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
connection-pool checkouts.
- **Histogram** — a distribution. Request latency, payload size. This is what
gives you percentiles.
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
read via a callback on collection (right for things you *poll*, like pool size).
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
nonsense. Use histograms.
- **Cardinality — the one thing that will bite you.** Every unique combination of
metric name + label values is a separate time series in memory. Add `account_id`
as a label on a 800-account demo and you've made 800 series; do it in production
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
~10 000 distinct values in production, it does not belong on a metric.** Put it on
a span. Histograms multiply this (one series per bucket per label combination), so
don't over-bucket either.
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
the concrete path), `http.status_code` class, `kind`, `currency`.
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
raw URL, timestamp, error message.
- **RED and USE.**
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
per-service dashboard.
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
connection pools, queues, disk. Saturation is the one everyone forgets and the
one that predicts collapse.
- You want both: RED tells you the service is slow, USE tells you which resource
is why.
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
and knowing both is the point:
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
`servicegraph` connectors read spans in flight and emit RED metrics and edge
metrics. This is what most blog posts (including the OneUptime article) show.
- *In the trace store:* Tempo's **metrics-generator** does the same job after
ingestion, remote-writing the result to Prometheus. **This is what your platform
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
`send_exemplars: true`, remote-writing to a Prometheus started with
`--web.enable-remote-write-receiver`.
- So RED metrics, the service map and exemplars **already exist** before you write
a single metric line. Do not add the collector connectors as well — you'd
duplicate every series under a second set of names. **Go find the real metric
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
they will not match the article's.
- The trade-off worth being able to argue: collector-side generation is cheaper
and works before storage, but sees only what passes through that collector;
Tempo-side generation sees everything that landed, survives collector restarts,
and can be enabled per-tenant — at the cost of load on the trace store.
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
click a spike on a latency graph and land in the exact trace. On your platform
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
`name: trace_id`. If you emit your own histograms with exemplars, the label must
be `trace_id` to match.
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
the pull model is why Prometheus has service discovery and why `up` exists.
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
events: a stable event name plus typed fields (`event="transaction_settled"`,
`account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
Consistent field names across all seven services — `account_id` everywhere, never
`accountId` in one place and `acct` in another. This is what makes LogQL queries
possible at all.
- **Trace–log correlation.** Every log line emitted inside an active span must carry
`trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
"one pane of glass": click a span → see its logs; click a log line → open its
trace. Both directions are configured, and you'll set both up.
- **Two ways to ship logs, and which one your platform actually does:**
- **stdout → collector scraping → Loki.** The container-native way. Your app just
prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
standard in Kubernetes. **This is the only path wired in your Alloy config**
(`loki.source.docker` reading the Docker socket).
- **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
with the same Resource attributes attached automatically. Cleaner correlation,
but the app owns delivery. **On your platform this currently goes nowhere** —
Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
dropped in silence.
- Do both anyway: use stdout for most services, and *extend Alloy* to wire the
OTLP logs path for one or two. Then argue the trade-off from experience. That
comparison — including the fact that a misconfigured pipeline fails silently
rather than loudly — is precisely what Babak will probe.
- **The field name is not a style choice here.** Your Loki datasource is provisioned
with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
A log line must carry lowercase `trace_id`. OpenTelemetry's logging
auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
logs and traces both arriving, no errors anywhere, and no clickable link. Hence
structlog with an explicit processor rather than the auto-injection shortcut.
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
your entire fleet, and Loki bills/indexes on volume.
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
`json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
redaction into the processor chain, not into each call site.
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
not log content. Labels should be few and low-cardinality (`service`, `env`,
`level`). Everything else lives in the log line body and is filtered at query time.
Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
in the JSON body and use a **derived field** to make it clickable.
### Resources
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
Read the first half. Best cardinality explanation available.
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
not a fact.
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
`servicegraph` connector config and its RED PromQL queries almost verbatim.
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
setup, including the hex formatting for trace/span IDs.
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
`OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
fields, if you'd rather not hand-roll a processor.
- Udemy backbone: the Loki + LogQL sections.
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
### Build
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
2. Custom application metrics, deliberately few:
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
- `ledgerline.fraud.score` (Histogram; no labels)
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
metrics, so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add them to
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
playbook. Prometheus scrapes your services directly. Note the irony and the
lesson — you're back to a pull model and a second client library, which is
exactly the fragmentation OTel exists to fix.
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
Alloy so app metrics flow through the collector as the architecture drawing
intends. This also closes a real diagram-vs-code gap in the repo.
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
all three signals.
Do the first to get moving; do one of the others afterwards and write up which
you'd choose for a real platform team and why. That write-up is a Phase 9
artefact.
4. Deliberately break it: add `account_id` as a label to
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
you'll never forget.
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
one. Understand why they differ slightly and when you'd want each.
6. **Second migration of the call-counting assertions.** The three tests you moved
to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
TraceQL query for `sanctions` client spans within a single trace answers
"how many lookups did this one transaction make?" directly, which is the
question the test was always really asking. Do it and compare all three
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
clearest possible demonstration of what each signal is actually for, on a
problem you already understand.
1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
processor that pulls the current span context and injects `trace_id` / `span_id`.
2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
handler: request received (or rely on the span), business decision, error.
3. Add a redaction processor with a denylist of field names.
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
with a `container` label (Alloy's relabel rule sets it from the Docker container
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
the receiver shows traffic, and the pipeline shows no logs output. Now extend
`config.alloy.j2` — add a `logs` output on the receiver plus an
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
That is the most transferable half-hour in the whole project.
6. Confirm the provisioned correlation works in both directions: Loki's
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
are verifying, not writing. If a link doesn't appear, the regex didn't match
your field name — check that before anything else.
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
(`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
in `docker compose logs` — the only cross-service call counter that existed at
Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
with LogQL over structured fields instead. This is a small change and a
pointed one: notice how much shorter and more precise the assertion gets, and
that it now works without shelling out to Docker at all. That delta is the
value of structured logging, measured on your own code.
**Done when:** you have a RED view per service, a working service map, and you can
click a latency spike through to a trace via an exemplar.
**Done when:** from a slow trace you can click into that request's logs, and from
any log line you can click into its trace — in both directions, in two clicks.
docs/roadmap/phase-5.md
+109 -59
View File
@@ -1,76 +1,126 @@
# Phase 5 — Dashboards and alerts as code
# Phase 5 — Metrics
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
**Status:** Not started
**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
dashboarding.
**Message:** Not blocked technically (metrics don't strictly need tracing
first), but sequenced after Phase 4 per the roadmap's ordering.
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
recreate in seconds.*
**Time-box:** 2 evenings.
### Concepts
- Three levels of "as code", and you should know when each applies:
1. **File provisioning** — YAML in `provisioning/datasources/` and
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
container. Grafana loads them at startup. Simplest, no state, no API tokens,
perfect for a Compose demo. **This alone satisfies the requirement.**
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
of Grafonnet (which is legacy and unofficial).
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
when this lands on your Kubernetes cluster.
- **Dashboards should answer questions, not display data.** A good service dashboard
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
is the pressure. Panels that nobody looks at during an incident are cost, not value.
- **Alerts as code too.** Grafana alert rules, contact points and notification
policies all provision from YAML. And keep the *alert definition* next to the
dashboard that explains it.
- **Instrument types.**
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
connection-pool checkouts.
- **Histogram** — a distribution. Request latency, payload size. This is what
gives you percentiles.
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
read via a callback on collection (right for things you *poll*, like pool size).
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
nonsense. Use histograms.
- **Cardinality — the one thing that will bite you.** Every unique combination of
metric name + label values is a separate time series in memory. Add `account_id`
as a label on a 800-account demo and you've made 800 series; do it in production
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
~10 000 distinct values in production, it does not belong on a metric.** Put it on
a span. Histograms multiply this (one series per bucket per label combination), so
don't over-bucket either.
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
the concrete path), `http.status_code` class, `kind`, `currency`.
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
raw URL, timestamp, error message.
- **RED and USE.**
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
per-service dashboard.
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
connection pools, queues, disk. Saturation is the one everyone forgets and the
one that predicts collapse.
- You want both: RED tells you the service is slow, USE tells you which resource
is why.
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
and knowing both is the point:
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
`servicegraph` connectors read spans in flight and emit RED metrics and edge
metrics. This is what most blog posts (including the OneUptime article) show.
- *In the trace store:* Tempo's **metrics-generator** does the same job after
ingestion, remote-writing the result to Prometheus. **This is what your platform
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
`send_exemplars: true`, remote-writing to a Prometheus started with
`--web.enable-remote-write-receiver`.
- So RED metrics, the service map and exemplars **already exist** before you write
a single metric line. Do not add the collector connectors as well — you'd
duplicate every series under a second set of names. **Go find the real metric
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
they will not match the article's.
- The trade-off worth being able to argue: collector-side generation is cheaper
and works before storage, but sees only what passes through that collector;
Tempo-side generation sees everything that landed, survives collector restarts,
and can be enabled per-tenant — at the cost of load on the trace store.
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
click a spike on a latency graph and land in the exact trace. On your platform
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
`name: trace_id`. If you emit your own histograms with exemplars, the label must
be `trace_id` to match.
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
the pull model is why Prometheus has service discovery and why `up` exists.
### Resources
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
example.
- Udemy backbone: dashboard design sections (panel types, transformations,
variables, dashboard best practice).
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
Read the first half. Best cardinality explanation available.
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
not a fact.
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
`servicegraph` connector config and its RED PromQL queries almost verbatim.
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
### Build
Your platform provisions **datasources only** — there is no dashboards provider at
all. So this phase is entirely yours, and the right home for it is inside the
Ansible role, using the same mechanism as everything else.
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
2. Custom application metrics, deliberately few:
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
- `ledgerline.fraud.score` (Histogram; no labels)
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
metrics, so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add them to
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
playbook. Prometheus scrapes your services directly. Note the irony and the
lesson — you're back to a pull model and a second client library, which is
exactly the fragmentation OTel exists to fix.
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
Alloy so app metrics flow through the collector as the architecture drawing
intends. This also closes a real diagram-vs-code gap in the repo.
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
all three signals.
Do the first to get moving; do one of the others afterwards and write up which
you'd choose for a real platform team and why. That write-up is a Phase 10
artefact.
4. Deliberately break it: add `account_id` as a label to
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
you'll never forget.
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
one. Understand why they differ slightly and when you'd want each.
6. **Second migration of the call-counting assertions.** The three tests you moved
to LogQL in Phase 4 can now count **client spans** in Tempo instead — a
TraceQL query for `sanctions` client spans within a single trace answers
"how many lookups did this one transaction make?" directly, which is the
question the test was always really asking. Do it and compare all three
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
clearest possible demonstration of what each signal is actually for, on a
problem you already understand.
1. **Extend the role.** Add
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
(a file-based dashboard provider pointing at a mounted directory), a
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
and copy them. The Compose template already mounts
`./config/grafana/provisioning` read-only, so the provider is picked up on
restart.
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
Foundation SDK (Python)**: one function producing a RED dashboard for a given
service name, looped over the seven services; a second producing a system
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
the generator *and* its output; the Ansible role ships the output.
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
than a local Makefile target, because it's the same command that builds the
platform.
4. While you're in there, review the provisioned datasources and add what's missing:
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
understand why `filterByTraceID: true` alone is sufficient here, and when it
wouldn't be.
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
**Done when:** re-running the playbook restores every dashboard from scratch, and no
dashboard has ever been saved through the Grafana UI.
**Done when:** you have a RED view per service, a working service map, and you can
click a latency spike through to a trace via an exemplar.
docs/roadmap/phase-6.md
+60 -78
View File
@@ -1,94 +1,76 @@
# Phase 6 — Load testing with k6, and percentiles
# Phase 6 — Dashboards and alerts as code
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
**Status:** Not started
**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
benefits from Phase 5's dashboards existing.
**Message:** Depends on Phases 3–5 producing traces, logs and metrics worth
dashboarding.
**Time-box:** 1–2 evenings.
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
recreate in seconds.*
### Concepts — P95 and P99 properly
### Concepts
You asked about these specifically, and there's more to them than "the 95th
percentile."
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
is.
- **You cannot average percentiles.** Not across instances, not across time buckets.
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
to aggregate the *histogram buckets* and compute the quantile from the sum:
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
bracket your actual SLO threshold. (Native histograms largely solve this and are
worth knowing about.)
- **Which population?** An overall p99 across all endpoints is dominated by whichever
endpoint has the most traffic. Slice by route, or you'll miss that
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
- **Tail latency amplification** — the most important idea here for a microservice
system. If one user request fans out to 10 backend calls, each with an independent
p99 of 1 s, then the chance the *user request* touches at least one slow call is
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
measurable — go measure it.
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
that.
- **Coordinated omission** — a load-generator trap. If your generator waits for a
response before sending the next request, then when the system stalls, the
generator stalls too and simply *doesn't record* the requests that would have been
slow. Your measured latency is systematically optimistic. Fix: use an **open
model** — a fixed *arrival rate* independent of response times. In k6 that means
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
`constant-vus`. Knowing this is a genuine senior-level differentiator.
### Concepts — k6
- **Test types:** smoke (does it work at all), average-load (normal traffic),
stress (past normal, find the knee), spike (sudden surge, test recovery),
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
fail the run; **thresholds are SLOs expressed in code and they set the process exit
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
view and the application's view sit on the same Grafana time axis. That side-by-side
is the single most convincing screenshot you can show Babak.
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
writing 60 lines of it.
- Three levels of "as code", and you should know when each applies:
1. **File provisioning** — YAML in `provisioning/datasources/` and
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
container. Grafana loads them at startup. Simplest, no state, no API tokens,
perfect for a Compose demo. **This alone satisfies the requirement.**
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
of Grafonnet (which is legacy and unofficial).
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
when this lands on your Kubernetes cluster.
- **Dashboards should answer questions, not display data.** A good service dashboard
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
is the pressure. Panels that nobody looks at during an incident are cost, not value.
- **Alerts as code too.** Grafana alert rules, contact points and notification
policies all provision from YAML. And keep the *alert definition* next to the
dashboard that explains it.
### Resources
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
*Thresholds*. Free, genuinely well written; ~2 h to work through.
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
hands-on).
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
- Grafana docs → *k6 output to Prometheus remote write*.
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
example.
- Udemy backbone: dashboard design sections (panel types, transformations,
variables, dashboard best practice).
### Build
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
sharing a `common.js` for setup and payload generation against random accounts.
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
`http_req_failed rate<0.01`.
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
sets. This before/after pair is your baseline for Phase 8.
6. Optional but impressive: have k6 send W3C trace context headers so load-test
requests appear as traces, letting you click from a k6 spike into a specific
trace. Check the current k6 docs for the tracing module's status — it has moved
between experimental and stable.
Your platform provisions **datasources only** — there is no dashboards provider at
all. So this phase is entirely yours, and the right home for it is inside the
Ansible role, using the same mechanism as everything else.
**Done when:** `make load-stress` produces a Grafana view where you can see the
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
service that broke first.
1. **Extend the role.** Add
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
(a file-based dashboard provider pointing at a mounted directory), a
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
and copy them. The Compose template already mounts
`./config/grafana/provisioning` read-only, so the provider is picked up on
restart.
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
Foundation SDK (Python)**: one function producing a RED dashboard for a given
service name, looped over the seven services; a second producing a system
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
the generator *and* its output; the Ansible role ships the output.
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
than a local Makefile target, because it's the same command that builds the
platform.
4. While you're in there, review the provisioned datasources and add what's missing:
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
understand why `filterByTraceID: true` alone is sufficient here, and when it
wouldn't be.
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.
**Done when:** re-running the playbook restores every dashboard from scratch, and no
dashboard has ever been saved through the Grafana UI.
docs/roadmap/phase-7.md
+78 -75
View File
@@ -1,91 +1,94 @@
# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
# Phase 7 — Load testing with k6, and percentiles
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
**Status:** Not started
**Message:** Depends on Phase 6's load-test baseline (the latency SLO should
come from measured numbers, not be picked from thin air).
**Message:** Depends on Phase 5 (metrics to plot k6's output against) and
benefits from Phase 6's dashboards existing.
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
**Time-box:** 1–2 evenings.
### Concepts
### Concepts — P95 and P99 properly
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
`POST /api/transactions` requests that returned 2xx within 500 ms.*
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
requests succeed within 500 ms, measured over a rolling 28 days."
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
Always **looser** than your SLO, so you have room to miss internally without
breaching commercially. If your SLA and SLO are the same number, you've made a
mistake.
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
budget; over 30 days that's about **43 minutes**. The budget is the thing that
turns reliability from an argument into arithmetic: if there's budget left, ship
features and take risks; if it's spent, the policy says you stop feature work and
spend on reliability. That policy — agreed in advance by dev, ops and product — is
the whole point.
- **Burn rate.** How fast you're consuming budget relative to the window.
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
is what stops you from either paging on every blip or noticing the slow bleed a
month late.
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
and it's worth being precise because interviewers probe exactly here.
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
from failure to restored service. It drives your architecture: replication,
warm standby, failover automation, runbook quality.
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
in time: how far back can you afford to lose. It drives your **backup and
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
every 15 minutes.
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
5 minutes of transactions." Both feed the SLA.
- Both are **business decisions, not technical ones** — you derive them by asking
what an hour of downtime or an hour of lost data actually costs, then buying the
cheapest architecture that meets it. Tightening either is expensive and
non-linear.
You asked about these specifically, and there's more to them than "the 95th
percentile."
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
is.
- **You cannot average percentiles.** Not across instances, not across time buckets.
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
to aggregate the *histogram buckets* and compute the quantile from the sum:
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
bracket your actual SLO threshold. (Native histograms largely solve this and are
worth knowing about.)
- **Which population?** An overall p99 across all endpoints is dominated by whichever
endpoint has the most traffic. Slice by route, or you'll miss that
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
- **Tail latency amplification** — the most important idea here for a microservice
system. If one user request fans out to 10 backend calls, each with an independent
p99 of 1 s, then the chance the *user request* touches at least one slow call is
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
measurable — go measure it.
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
that.
- **Coordinated omission** — a load-generator trap. If your generator waits for a
response before sending the next request, then when the system stalls, the
generator stalls too and simply *doesn't record* the requests that would have been
slow. Your measured latency is systematically optimistic. Fix: use an **open
model** — a fixed *arrival rate* independent of response times. In k6 that means
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
`constant-vus`. Knowing this is a genuine senior-level differentiator.
### Concepts — k6
- **Test types:** smoke (does it work at all), average-load (normal traffic),
stress (past normal, find the knee), spike (sudden surge, test recovery),
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
fail the run; **thresholds are SLOs expressed in code and they set the process exit
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
view and the application's view sit on the same Grafana time axis. That side-by-side
is the single most convincing screenshot you can show Babak.
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
writing 60 lines of it.
### Resources
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
all free PDFs. Run the exercises solo with Ledgerline as the subject.
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
free, self-paced, deeper.
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
overview; pair it with your Postgres backup knowledge.
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
*Thresholds*. Free, genuinely well written; ~2 h to work through.
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
hands-on).
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
- Grafana docs → *k6 output to Prometheus remote write*.
### Build
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
- Availability SLI/SLO on `POST /api/transactions`
- Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
from thin air)
- A freshness/lag SLI on the notifier consumer group
- A stated SLA that is deliberately looser than each SLO
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
- An error budget policy: what happens at 50%, 75% and 100% consumed
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
be able to argue the alternative: Grafana unified alerting can evaluate across
datasources (including Loki queries) and lives with your dashboards, whereas
Alertmanager is metrics-only but is the ecosystem standard and gives you
grouping, deduplication, inhibition and silences for free. Note which you chose
and why in `docs/SLO.md`.
3. Add an error-budget panel to the overview dashboard: budget remaining, current
burn rate, projected exhaustion.
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
watch the fast-burn alert fire.
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
sharing a `common.js` for setup and payload generation against random accounts.
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
`http_req_failed rate<0.01`.
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
sets. This before/after pair is your baseline for Phase 9.
6. Optional but impressive: have k6 send W3C trace context headers so load-test
requests appear as traces, letting you click from a k6 spike into a specific
trace. Check the current k6 docs for the tracing module's status — it has moved
between experimental and stable.
**Done when:** you can explain, in one paragraph each and without notes, the
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
decision.
**Done when:** `make load-stress` produces a Grafana view where you can see the
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
service that broke first.
docs/roadmap/phase-8.md
+75 -74
View File
@@ -1,90 +1,91 @@
# Phase 8 — Bottleneck hunting (the capstone)
# Phase 8 — SLI, SLO, SLA, error budgets, RTO, RPO
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
**Status:** Not started
**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
**Message:** Depends on Phase 7's load-test baseline (the latency SLO should
come from measured numbers, not be picked from thin air).
**Time-box:** 2 evenings. This is where the whole thing pays off.
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
### Concepts — how to actually answer "how do I find bottlenecks?"
### Concepts
Babak asked how you'd *think* about this, not for a list of tools. Here's the
method, roughly in order. The discipline is to go **top-down from the user, not
bottom-up from the infrastructure** — bottom-up is how people spend three hours
tuning a database that wasn't the problem.
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
2. **Localise to a service** using RED + the service graph. Which node's duration
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
the graph from the leaves inward.
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
several slow traces plus one fast trace of the same endpoint as a control.
4. **Read the trace for five specific shapes:**
- **A long span** — the work itself is slow. Look at its attributes: which query,
which host, which parameters.
- **A gap** — wall-clock time not covered by any span. This is the interesting
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
blocked event loop, or a missing instrumentation library. Ledgerline's
`pool_starvation` and `blocking_event_loop` both show up as gaps.
- **Serialisation** — spans laid end to end that could have been concurrent.
Ledgerline's `no_batching` looks exactly like this.
- **Repetition** — the same span name 150 times in a row. That's your N+1.
- **Retries** — the same client span repeated 3–4 times with errors. That's
`retry_storm`, and note that it makes an *overload* worse, never better.
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
memory. A resource at 100% utilisation with a growing queue is saturated;
latency there is a *symptom*, not the cause.
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
concurrent slots. If your pool is 5, you queue — and queueing latency grows
non-linearly as you approach saturation. This one formula explains most
connection-pool and worker-count incidents you'll ever see.
7. **Check for tail amplification** (Phase 6) before blaming any single service.
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
over the *same duration*. One variable at a time. Never declare a fix from a
dashboard that "looks better."
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
becomes the bottleneck. Re-measure after every change; stop when you're inside
the SLO, not when you run out of ideas.
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
or lock waits inside Postgres, or which line of Python burned the cycles. That's
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
`POST /api/transactions` requests that returned 2xx within 500 ms.*
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
requests succeed within 500 ms, measured over a rolling 28 days."
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
Always **looser** than your SLO, so you have room to miss internally without
breaching commercially. If your SLA and SLO are the same number, you've made a
mistake.
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
budget; over 30 days that's about **43 minutes**. The budget is the thing that
turns reliability from an argument into arithmetic: if there's budget left, ship
features and take risks; if it's spent, the policy says you stop feature work and
spend on reliability. That policy — agreed in advance by dev, ops and product — is
the whole point.
- **Burn rate.** How fast you're consuming budget relative to the window.
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
is what stops you from either paging on every blip or noticing the slow bleed a
month late.
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
and it's worth being precise because interviewers probe exactly here.
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
from failure to restored service. It drives your architecture: replication,
warm standby, failover automation, runbook quality.
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
in time: how far back can you afford to lose. It drives your **backup and
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
every 15 minutes.
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
5 minutes of transactions." Both feed the SLA.
- Both are **business decisions, not technical ones** — you derive them by asking
what an hour of downtime or an hour of lost data actually costs, then buying the
cheapest architecture that meets it. Tightening either is expensive and
non-linear.
### Resources
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
for saturation.
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
thing ever written on why microservice latency behaves the way it does.
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
these; connect them to the trace's DB spans.
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
all free PDFs. Run the exercises solo with Ledgerline as the subject.
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
free, self-paced, deeper.
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
overview; pair it with your Postgres backup knowledge.
### Build
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
`docs/PATHOLOGIES.md`.**
2. Run `make load-average` for 10 minutes.
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
like a real incident doc: what you observed, what you hypothesised, what you
checked, what it turned out to be. Include screenshots.
4. When you think you've found them all, run `make chaos-clean`, enable them one at
a time, and confirm each signature.
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
6. Repeat once with a new random set. The second run is where the method actually
becomes yours.
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
- Availability SLI/SLO on `POST /api/transactions`
- Latency SLI/SLO on the same (pick the threshold from your Phase 7 baseline, not
from thin air)
- A freshness/lag SLI on the notifier consumer group
- A stated SLA that is deliberately looser than each SLO
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
- An error budget policy: what happens at 50%, 75% and 100% consumed
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
be able to argue the alternative: Grafana unified alerting can evaluate across
datasources (including Loki queries) and lives with your dashboards, whereas
Alertmanager is metrics-only but is the ecosystem standard and gives you
grouping, deduplication, inhibition and silences for free. Note which you chose
and why in `docs/SLO.md`.
3. Add an error-budget panel to the overview dashboard: budget remaining, current
burn rate, projected exhaustion.
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
watch the fast-burn alert fire.
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
**Done when:** you can explain, in one paragraph each and without notes, the
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
decision.
docs/roadmap/phase-9.md
+74 -145
View File
@@ -1,161 +1,90 @@
# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
# Phase 9 — Bottleneck hunting (the capstone)
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
**Status:** Not started
**Message:** Not technically blocked by earlier phases (it stands on its own
as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
re-diagnose in Kibana already exists.
**Message:** Depends on Phases 3–8 all being in place (traces, logs, metrics,
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
**Time-box:** 1–2 evenings.
**Time-box:** 2 evenings. This is where the whole thing pays off.
Babak's remaining questions are all *judgment* questions. He isn't testing whether
you can recite what Filebeat does; he's testing whether you have a framework for
choosing. Here's the framework, then the answers as worked examples.
### Concepts — how to actually answer "how do I find bottlenecks?"
### The framework — six axes to reason on
Babak asked how you'd *think* about this, not for a list of tools. Here's the
method, roughly in order. The discipline is to go **top-down from the user, not
bottom-up from the infrastructure** — bottom-up is how people spend three hours
tuning a database that wasn't the problem.
Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
1. **Where does the data originate?** Inside your application (you can instrument it)
or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
2. **What shape is it already in?** Structured JSON at the source, or unstructured
text needing grok/regex parsing? Parsing cost has to land *somewhere*.
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
4. **What's the destination, and how many?** One backend forever, or several, or one
you expect to change?
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
on every host) distributes cost and adds resilience but multiplies config surface.
Parsing centrally concentrates cost and creates a single point of failure — but
one place to change rules.
6. **What already exists, and what does it cost?** Existing Elastic investment,
existing licences, team skills, and licensing constraints (Elastic's Filebeat and
Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
Bit and Vector are permissively licensed). Migration is never free.
### The answers
**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
from a host or container and forwards them with minimal transformation — and your
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
included. Reach for it when the source is a file you don't control the format of, the
app can't be modified, and Elasticsearch is where it's going.
**When to use Logstash.** When you need real *transformation*: parsing unstructured
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
splitting or merging events, conditional routing to multiple destinations, or
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
per host.
**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
architecture and the reason is a **division of labour between edge and centre**:
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
backpressure, ships onward. You want the thing running in a thousand places to be
small and boring.
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
you want them in a handful of places you can redeploy, not on a thousand hosts.
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
persistent queue absorbs it and Filebeat backs off gracefully, rather than every
host queuing independently or dropping.
So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
need central buffering. Skip Logstash when your logs are already structured JSON and
Elasticsearch ingest pipelines can do the little parsing that remains.
**When to use OpenTelemetry.**
- You own the application code and want telemetry generated *from inside it* — spans
with business context, custom metrics, logs already carrying trace IDs.
- You need **more than one signal**, correlated. This is the strongest argument: one
SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
threaded through all three. Neither Filebeat nor Logstash does traces.
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
data model across all of them.
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
Jaeger by editing Collector YAML.
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
rewriting — without touching the app.
**When *not* to use OpenTelemetry.** This is the question that separates people who
have used it from people who have read about it.
- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
file-based collector.
- **You only need one signal from one source with no correlation.** A single nginx
access log going to a single log store: OTel is more machinery than the job needs.
Fluent Bit or Filebeat is simpler and finishes the job today.
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
Elastic estate with hundreds of Beats modules, dashboards and detection rules is
years of investment. "We should use OTel because it's the standard" is not a
business case. Note the direction of travel though: the Collector can already
receive from and export to most of these, so a *gradual* migration is usually the
right shape — often by putting a Collector or Vector in front of the existing
agents rather than ripping them out.
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
retention, different query patterns, different cost model.
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
matured at different rates per language, and profiling is still alpha. Check the
status page for your language before betting on a signal.
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
on it. A two-person team that needs error tracking by Friday should install Sentry.
The senior framing to give Babak: *these tools aren't competitors so much as
different layers.* OpenTelemetry is a **generation and transport standard** for
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
Elasticsearch are **stores**. Most real architectures use one from each layer, and
the interesting question is always which layer a given problem lives in.
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
2. **Localise to a service** using RED + the service graph. Which node's duration
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
the graph from the leaves inward.
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
several slow traces plus one fast trace of the same endpoint as a control.
4. **Read the trace for five specific shapes:**
- **A long span** — the work itself is slow. Look at its attributes: which query,
which host, which parameters.
- **A gap** — wall-clock time not covered by any span. This is the interesting
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
blocked event loop, or a missing instrumentation library. Ledgerline's
`pool_starvation` and `blocking_event_loop` both show up as gaps.
- **Serialisation** — spans laid end to end that could have been concurrent.
Ledgerline's `no_batching` looks exactly like this.
- **Repetition** — the same span name 150 times in a row. That's your N+1.
- **Retries** — the same client span repeated 3–4 times with errors. That's
`retry_storm`, and note that it makes an *overload* worse, never better.
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
memory. A resource at 100% utilisation with a growing queue is saturated;
latency there is a *symptom*, not the cause.
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
concurrent slots. If your pool is 5, you queue — and queueing latency grows
non-linearly as you approach saturation. This one formula explains most
connection-pool and worker-count incidents you'll ever see.
7. **Check for tail amplification** (Phase 7) before blaming any single service.
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
over the *same duration*. One variable at a time. Never declare a fix from a
dashboard that "looks better."
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
becomes the bottleneck. Re-measure after every change; stop when you're inside
the SLO, not when you run out of ideas.
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
or lock waits inside Postgres, or which line of Python burned the cycles. That's
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
### Resources
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
read these to see how the Collector does the Filebeat/Logstash job itself.
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
caveats.
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
for saturation.
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
thing ever written on why microservice latency behaves the way it does.
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
these; connect them to the trace's DB spans.
### Build — and this is no longer hypothetical
### Build
Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
Filebeat reading Docker container logs and Logstash parsing JSON into daily
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
questions — the codebase he handed you contains both worlds.
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
`docs/PATHOLOGIES.md`.**
2. Run `make load-average` for 10 minutes.
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
like a real incident doc: what you observed, what you hypothesised, what you
checked, what it turned out to be. Include screenshots.
4. When you think you've found them all, run `make chaos-clean`, enable them one at
a time, and confirm each signature.
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
6. Repeat once with a new random set. The second run is where the method actually
becomes yours.
1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
Alloy → Loki. Compare them on the same data.
2. Test the direct ingest path the README documents — pipe a JSON line into
Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
parsing happens before you change it.
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
that parses them. Then do the same job with Alloy's `loki.process` stages. Time
both. Note which one you'd rather debug at 2 a.m.
4. Compare the *investigation experience* directly: take one incident you already
diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
for some questions, and Loki's cost model genuinely beats Elasticsearch's for
others.
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
permissively licensed. For some employers that alone decides it.
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
valuably — what you actually found annoying about each. Having run both stacks
against the same application is the difference between reciting a comparison and
arguing one.
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
docs/roadmap/summary.md
+28 -24
View File
@@ -6,7 +6,7 @@ for the phase you're actually working on. This file exists so an agent (or
you) can check overall status and sequencing without loading every phase's
full detail.*
Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase
Twelve phases (0 through 11 — Phase 0 has two halves, 0a and 0b). Each phase
file follows the same shape: **Status**, **Message** (what's done, notes,
warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
@@ -16,56 +16,60 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|---|---|---|---|---|---|
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
| 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
| 10 | [`phase-10.md`](phase-10.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
| 11 | [`phase-11.md`](phase-11.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
## What each phase is, in one line
- **0 — Build the base project, test-first.** Write the black-box test suite,
then implement the seven-service system against it as a frozen contract.
**Done** — see `phase-0.md` for the full implementation report.
- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
- **1 — Deployment.** Put Ledgerline on a server with Docker Compose behind
Traefik and a Let's Encrypt certificate, served at a real domain. Plumbing
only — no telemetry, no app changes. **Done** — see `phase-1.md`.
- **2 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
Ansible and get one auto-instrumented trace into Tempo.
- **2 — Distributed tracing.** The heart of the task: auto- and
- **3 — Distributed tracing.** The heart of the task: auto- and
manual-instrument all seven services, and hand-propagate trace context
across the Redis Stream hop, which carries none by default.
- **3 — Structured logs, correlated to traces.** Replace prose logging with
- **4 — Structured logs, correlated to traces.** Replace prose logging with
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
in both directions.
- **4 — Metrics.** Instrument RED metrics per service, understand cardinality
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
silently. Tempo's metrics-generator already produces span metrics and the
service graph — don't duplicate it in the collector.
- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
Grafana Foundation SDK and ship it through the Ansible role, so "destroy
and recreate in seconds" is a real, repeatable command.
- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't
- **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
be averaged, what coordinated omission is, and drive Ledgerline with
arrival-rate executors to get a clean before/after load profile.
- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
SLOs from the Phase 6 baseline (not from thin air), write an error-budget
- **8 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
SLOs from the Phase 7 baseline (not from thin air), write an error-budget
policy, and wire multi-burn-rate alerts.
- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
- **9 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
of chaos flags, find the pathologies from telemetry alone using a top-down,
hypothesis-driven method, and write it up as an incident doc.
- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
- **10 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
OpenTelemetry on six concrete axes, using the platform's real second stack
(Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
- **10 — Package the delivery.** README, a rehearsed five-minute demo, the
- **11 — Package the delivery.** README, a rehearsed five-minute demo, the
written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`,
`CONCEPTS.md`), and an honest "what I'd do differently in production"
section.
## If you have to cut scope
Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not
cut Phase 8** — it's the only phase that proves you learned anything.
Cut Phase 10's build (keep the writing) and Phase 7's soak test first. **Do not
cut Phase 9** — it's the only phase that proves you learned anything.