Add project documentation and Claude instructions
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
This commit is contained in:
Movses Aslanyan
@@ -0,0 +1,377 @@
|
||||
# Observability & APM — Delivery Roadmap for Babak's Task
|
||||
|
||||
*Part of `docs/roadmap/` — platform background and reading guidance shared by
|
||||
every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`;
|
||||
[`summary.md`](summary.md) gives the one-page overview and status of all of
|
||||
them.*
|
||||
|
||||
Ten phases. Each one: concepts first, then the build, then a checkable exit
|
||||
condition. Total time-boxed at roughly **4–6 weeks part-time** (evenings +
|
||||
weekends). Nothing here is theory for theory's sake — every concept block is
|
||||
followed immediately by the place it lands in the code.
|
||||
|
||||
---
|
||||
|
||||
## Read this first — the four questions blocking you
|
||||
|
||||
### "What *is* an APM? Do you install it, use it, implement it, or build it?"
|
||||
|
||||
APM (Application Performance Monitoring) is a **product category and a capability**,
|
||||
not a protocol or a single tool. A system has APM when you can answer, for
|
||||
production traffic: what's the request rate, error rate and latency distribution
|
||||
per endpoint; which services call which; where did *this specific* request spend its
|
||||
time; which database query or external call was slow; and what error was thrown,
|
||||
with the stack.
|
||||
|
||||
The verb depends on the vendor model:
|
||||
|
||||
- **Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM).** You
|
||||
*install* an agent that attaches to your runtime and ships to their backend. Mostly
|
||||
a deployment exercise. Turnkey, expensive, proprietary.
|
||||
- **Grafana stack — what Babak asked for.** There is no product called "Grafana APM"
|
||||
that you install. You **assemble** the capability from four layers:
|
||||
|
||||
| Layer | What you do | Component |
|
||||
|---|---|---|
|
||||
| Instrumentation | *Implement* — this is code you write | OpenTelemetry SDK inside your services |
|
||||
| Pipeline | *Deploy + configure* | OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution) |
|
||||
| Storage | *Deploy* | Tempo (traces), Loki (logs), Prometheus or Mimir (metrics) |
|
||||
| Query & UI | *Configure* | Grafana + provisioned datasources with cross-links |
|
||||
|
||||
So: you **implement** instrumentation, you **deploy** the pipeline and stores, you
|
||||
**configure** the correlation. The "APM" is the emergent property of all four
|
||||
working together.
|
||||
|
||||
The distinction that actually matters: APM data is generated *inside* the
|
||||
application, keyed to *individual requests*, and *correlated* across signals.
|
||||
Infrastructure monitoring (CPU, disk, pod restarts) is generated *outside* the
|
||||
application and aggregated. You need both; only the first is APM.
|
||||
|
||||
### "What's the difference between Logs, Metrics, Events and Traces?"
|
||||
|
||||
The clean way to hold this in your head is: **an event is the atom; the other three
|
||||
are shapes you give it.**
|
||||
|
||||
| | What it is | Cost shape | Answers |
|
||||
|---|---|---|---|
|
||||
| **Event** | A structured record of one thing that happened, at one instant, with arbitrary attributes attached | Grows with traffic × attribute count | "What exactly happened here?" |
|
||||
| **Log** | An event rendered for storage/search. A *structured* log is literally an event; an unstructured log is an event you threw the schema away from | Grows with traffic | "Why did it fail?" |
|
||||
| **Metric** | A pre-aggregated numeric measurement over many events, sliced by a small set of labels | Grows with *label cardinality*, ~flat in traffic | "Is it broken? How much? Since when?" |
|
||||
| **Trace** | A causally linked tree of events that each have a *duration* and a parent — a span tree spanning process boundaries | Grows with traffic × span count | "Where did the time go, across services?" |
|
||||
|
||||
Three consequences worth internalising now:
|
||||
|
||||
1. **Metrics are cheap in traffic but expensive in cardinality.** Doubling your
|
||||
request rate barely changes Prometheus's memory; adding `user_id` as a label
|
||||
can kill it. Logs and traces are the opposite: cheap in dimensions, expensive
|
||||
in volume. This asymmetry drives every design decision later.
|
||||
2. **Traces are events with two extra fields:** a duration and a parent pointer.
|
||||
That's the whole trick. Everything else (service maps, critical paths, span
|
||||
metrics) is derived from those two fields.
|
||||
3. In OpenTelemetry, "events" aren't a fourth top-level signal you export
|
||||
separately. They show up as **span events** (a timestamped annotation attached
|
||||
to a span — this is how `record_exception` works) and as **log records carrying
|
||||
an `event.name` attribute**; the Events API is built on the Logs data model.
|
||||
When Babak says M.E.L.T., he most likely means "logs plus the discrete
|
||||
business/domain events you care about" — check with him, but treat it as
|
||||
*structured logs and span events*, not a separate pipeline.
|
||||
|
||||
### "Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?"
|
||||
|
||||
**OpenTelemetry first, without much doubt.**
|
||||
|
||||
Loki, Tempo, Prometheus and Mimir are *destinations*. They're databases with query
|
||||
languages. OpenTelemetry is the **producer-side data model and wire protocol** — it's
|
||||
the only part of this that touches your Python code, and it's the part that
|
||||
determines what the destinations can possibly show you. If you learn Loki first you
|
||||
learn Loki's label model and its promtail-isms, most of which don't transfer. If you
|
||||
learn OTel first, every backend becomes "point OTLP at a different URL."
|
||||
|
||||
One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
|
||||
early, in parallel with OTel metrics — because it *constrains what you're allowed to
|
||||
emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
|
||||
(Phases 4–5), not up front.
|
||||
|
||||
Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
|
||||
part of the Grafana stack and you do not need them for this task. Phase 9 covers
|
||||
them as a judgment question, which is exactly how Babak framed them.
|
||||
|
||||
### Verdict on the OneUptime article you found
|
||||
|
||||
**Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
|
||||
teaching resource. Read it in **Phase 4/5**, not now; it assumes you already
|
||||
understand spans, span metrics and PromQL.
|
||||
|
||||
What's genuinely good in it, and rare to find in one place:
|
||||
|
||||
- The `spanmetrics` and `servicegraph` **connectors** in the Collector — this is how
|
||||
you get RED metrics and a service map "for free" from traces you already emit.
|
||||
This is the single highest-leverage config in the whole article.
|
||||
- The Tempo datasource provisioning YAML with `tracesToLogsV2`, `tracesToMetrics`,
|
||||
`serviceMap` and `nodeGraph` — the click-through correlation that makes Grafana
|
||||
feel like an APM.
|
||||
- `exemplarTraceIdDestinations` on the Prometheus datasource — jump from a latency
|
||||
spike on a graph straight into the trace that caused it.
|
||||
- The tail-sampling policy block (keep all errors, keep all slow traces, sample the
|
||||
rest).
|
||||
|
||||
Two caveats:
|
||||
|
||||
- Its Collector config exports logs via the **`loki` exporter**. Loki now accepts
|
||||
**OTLP natively** on its own endpoint, and that is the current recommended path —
|
||||
fewer moving parts, better attribute mapping. Use the native OTLP route.
|
||||
- It's a vendor blog (OneUptime sells a competing product). The Grafana content is
|
||||
accurate, but treat the official OpenTelemetry and Grafana docs as the source of
|
||||
truth for anything spec-level.
|
||||
- **Its `spanmetrics` / `servicegraph` Collector connectors do not apply to your
|
||||
stack.** See the next section — your platform generates those metrics in Tempo
|
||||
instead. The article's PromQL is still useful, but the metric names differ.
|
||||
|
||||
---
|
||||
|
||||
## The platform Babak gave you (`observability-ansible`)
|
||||
|
||||
This changes *where* your telemetry goes and *how the pipeline is configured*. It
|
||||
does not change the learning arc, the app, or the test suite. Read this before
|
||||
Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9.
|
||||
|
||||
### What the repo is
|
||||
|
||||
An Ansible project that deploys **two independent observability backends** on one
|
||||
Ubuntu/Debian host via Docker Compose:
|
||||
|
||||
1. **ELK + Grafana** — Filebeat → Logstash → Elasticsearch → Kibana/Grafana
|
||||
2. **Grafana stack** — Prometheus + Alertmanager + Loki + Tempo + Grafana, with
|
||||
**Grafana Alloy** as the collector and node-exporter for host metrics
|
||||
|
||||
It contains **no application code and no dashboards**. It is the *platform half* of
|
||||
Babak's task, handed to you pre-built. Your half is everything that produces
|
||||
telemetry and everything that reads it.
|
||||
|
||||
Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
|
||||
`playbooks/elk-grafana.yml`, idempotent, versions pinned in `group_vars/all.yml`
|
||||
(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
|
||||
Elastic 9.1.3). Everything lands under `/opt/observability/`.
|
||||
|
||||
### The architecture
|
||||
|
||||
Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
|
||||
**Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline →
|
||||
Explore & Correlate**, with three input classes feeding it.
|
||||
|
||||
```
|
||||
App metrics ──/metrics, Prometheus exposition──┐
|
||||
Container logs ──Docker socket, tail──▶ Alloy ──push API──▶ Loki
|
||||
Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo
|
||||
("spans + resource attrs") │
|
||||
Node Exporter :9100 ──────scrape──────┐ │
|
||||
▼ │
|
||||
Prometheus ◀──remote_write────────┘
|
||||
(TSDB, PromQL, rules) span metrics + exemplars
|
||||
│
|
||||
firing alerts ▼
|
||||
Alertmanager ──route──▶ Receivers
|
||||
(group, dedupe) (Slack, Email, PagerDuty,
|
||||
│ webhook, SMTP)
|
||||
Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four
|
||||
Correlation key: trace_id / exemplars
|
||||
```
|
||||
|
||||
Note what the diagram says about the OTLP box: **"spans + resource attrs"**. Traces
|
||||
only. And app metrics are labelled **"/metrics endpoint, Prometheus exposition"** —
|
||||
not OTLP. The diagram and the code agree on this; it is a design decision, not an
|
||||
oversight. Plan your instrumentation around it.
|
||||
|
||||
### One place the diagram and the code disagree
|
||||
|
||||
The diagram draws app metrics as `App metrics → scrape → Alloy → /metrics →
|
||||
Prometheus`. The shipped config does not do that:
|
||||
|
||||
- `config.alloy.j2` contains **no `prometheus.scrape` component at all** — Alloy
|
||||
collects logs and receives traces, nothing else.
|
||||
- `prometheus.yml.j2` scrapes `prometheus`, `node-exporter`, `loki`, `tempo`,
|
||||
`alloy:12345`, `grafana`, then loops over `prometheus_extra_scrape_jobs`. So your
|
||||
app targets are scraped by **Prometheus directly**, bypassing Alloy entirely. The
|
||||
green `/metrics` arrow from Alloy to Prometheus in the diagram is Prometheus
|
||||
scraping Alloy's **own** self-monitoring endpoint, not your app's metrics
|
||||
passing through it.
|
||||
|
||||
Neither is wrong; the diagram shows the intended architecture and the code
|
||||
implements a simpler one. Closing that gap — adding `prometheus.scrape` +
|
||||
`prometheus.remote_write` to Alloy so metrics really do flow through the collector —
|
||||
is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth
|
||||
mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
|
||||
exactly the skill this task is meant to build.
|
||||
|
||||
### Four things in it that will cost you a day if you don't know them
|
||||
|
||||
**1. Alloy's OTLP receiver only forwards traces.** The config is literally:
|
||||
|
||||
```
|
||||
otelcol.receiver.otlp "default" {
|
||||
grpc { endpoint = "0.0.0.0:4317" }
|
||||
http { endpoint = "0.0.0.0:4318" }
|
||||
output {
|
||||
traces = [otelcol.exporter.otlp.tempo.input]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
|
||||
your app sends are accepted and then **silently discarded** — no error, no warning,
|
||||
just nothing arriving in Grafana. This is the single most likely thing to eat an
|
||||
evening.
|
||||
|
||||
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
|
||||
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
|
||||
input. So you have three options, and you should understand all three:
|
||||
|
||||
- *Follow the platform's grain (simplest):* expose a Prometheus `/metrics` endpoint
|
||||
per service and register them in `prometheus_extra_scrape_jobs` in
|
||||
`group_vars/all.yml`. Prometheus scrapes your services directly. Logs continue to
|
||||
arrive via stdout scraping.
|
||||
- *Implement what the diagram draws:* add `prometheus.scrape` +
|
||||
`prometheus.remote_write` components to Alloy so app metrics flow through the
|
||||
collector, as the architecture intends. Buys you central relabeling and one place
|
||||
to change collection policy.
|
||||
- *Go all-OTLP:* add `metrics` and `logs` outputs to the receiver, plus
|
||||
`otelcol.exporter.prometheus` → `prometheus.remote_write` and
|
||||
`otelcol.exporter.loki` → `loki.write`. One protocol for all three signals — the
|
||||
architecture you'd argue for on a greenfield platform, and the furthest from what
|
||||
ships.
|
||||
|
||||
Do the first to get moving, then do at least one of the others as a deliberate
|
||||
exercise and write up the trade-off. That comparison is worth more than any single
|
||||
answer.
|
||||
|
||||
**2. Span metrics come from Tempo, not from the collector.** `tempo.yml.j2` enables:
|
||||
|
||||
```
|
||||
metrics_generator:
|
||||
storage:
|
||||
remote_write:
|
||||
- url: http://prometheus:9090/api/v1/write
|
||||
send_exemplars: true
|
||||
overrides:
|
||||
defaults:
|
||||
metrics_generator:
|
||||
processors: [service-graphs, span-metrics, local-blocks]
|
||||
```
|
||||
|
||||
So RED metrics, the service graph, and exemplars already exist — generated inside
|
||||
Tempo and remote-written into Prometheus (which runs with
|
||||
`--web.enable-remote-write-receiver`). **Do not add `spanmetrics`/`servicegraph`
|
||||
connectors to Alloy** — you'd be generating the same series twice under different
|
||||
names. The metric names differ from the OneUptime article's; go find the actual
|
||||
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
|
||||
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.)
|
||||
|
||||
**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
|
||||
datasource is provisioned with:
|
||||
|
||||
```
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
|
||||
name: TraceID
|
||||
```
|
||||
|
||||
Your log lines must contain **lowercase** `trace_id` (or `traceid` / `trace-id`).
|
||||
This matters more than it looks: OpenTelemetry's `opentelemetry-instrumentation-logging`
|
||||
auto-injection emits **`otelTraceID`**, which does **not** match that pattern.
|
||||
So on this platform, the structlog processor emitting a lowercase `trace_id` field
|
||||
isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus
|
||||
datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram states
|
||||
this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
|
||||
in the Explore & Correlate column hangs off that one field name.
|
||||
|
||||
**4. There is no dashboard provisioning at all.** The Grafana provisioning mount
|
||||
contains only `datasources/`. Every dashboard is yours to add — which means Babak's
|
||||
"create dashboards as code, destroy and recreate in seconds" requirement is entirely
|
||||
your deliverable, and the natural home for it is *inside this Ansible role*.
|
||||
|
||||
### What this changes, phase by phase
|
||||
|
||||
| Phase | Impact |
|
||||
|---|---|
|
||||
| 0 / 0a | **None.** App and test suite are unaffected. |
|
||||
| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
|
||||
| 2 | Endpoint changes to Alloy; tracing work itself is identical. |
|
||||
| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
|
||||
| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
|
||||
| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
|
||||
| 6 | None. |
|
||||
| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
|
||||
| 8 | None, plus TraceQL metrics from `local-blocks`. |
|
||||
| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
|
||||
| 10 | Demo now includes `ansible-playbook` rebuilding the platform. |
|
||||
|
||||
### Where to run it
|
||||
|
||||
The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
|
||||
ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside
|
||||
everything else will hurt. Use a separate VM (a cheap cloud instance or a local
|
||||
Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing
|
||||
at it. That also makes the deployment *real* rather than a localhost shortcut, which
|
||||
is the point of it being Ansible.
|
||||
|
||||
If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker
|
||||
socket and will scrape **every** container's stdout, including Ledgerline's,
|
||||
automatically — convenient, and worth knowing before you wonder where those logs
|
||||
came from.
|
||||
|
||||
### One thing to confirm with Babak
|
||||
|
||||
Is this repo a **platform you must consume as-is**, or a **starting point you may
|
||||
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
|
||||
assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
|
||||
extend `group_vars`. If Babak intends it as a fixed platform, several
|
||||
recommendations below change — ask before Phase 3.
|
||||
|
||||
---
|
||||
|
||||
## Your resource spine
|
||||
|
||||
Pick **one paid course as a backbone** and use the free material as targeted
|
||||
supplements. Both Udemy options below ship captions, which matters for your
|
||||
subtitle-to-notes workflow.
|
||||
|
||||
**Backbone — choose one:**
|
||||
|
||||
| Resource | Format | Length | Cost | Best for |
|
||||
|---|---|---|---|---|
|
||||
| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. |
|
||||
| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. |
|
||||
|
||||
If budget allows, buy both — they overlap by maybe 30% and the second half of each
|
||||
is different. If only one: take the **OpenTelemetry** one, because instrumentation
|
||||
is the part you have to *write*, and Grafana's own docs are excellent for the rest.
|
||||
|
||||
**Free, and worth your time:**
|
||||
|
||||
| Resource | Format | Length | Use in |
|
||||
|---|---|---|---|
|
||||
| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
|
||||
| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. |
|
||||
| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. |
|
||||
| **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
|
||||
| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
|
||||
| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. |
|
||||
| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 |
|
||||
| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
|
||||
| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. |
|
||||
| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on |
|
||||
| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 |
|
||||
| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. |
|
||||
| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
|
||||
| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 |
|
||||
| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. |
|
||||
| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 |
|
||||
| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 |
|
||||
|
||||
**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on
|
||||
your k8s cluster until everything works locally — debugging instrumentation and
|
||||
debugging Kubernetes at the same time is how these projects die. Porting it to the
|
||||
cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half
|
||||
Babak has planned for you.
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Phase 0 — Build the base project, test-first
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`intro.md`](intro.md) · Next: [`phase-1.md`](phase-1.md)*
|
||||
|
||||
**Status:** Done
|
||||
|
||||
**Message:** Both halves complete and verified against the live stack, not just
|
||||
collected.
|
||||
|
||||
- **0a (test suite):** 81 tests across `tests/` (11 files + a 5-module
|
||||
harness), covering all eleven goals G1–G11 and all 13 named pathologies.
|
||||
Verified genuinely red against the empty repo before any implementation
|
||||
existed; `tests/RED-BASELINE.md` records that run. `tests/README.md`
|
||||
documents the invented request-contract fields, measurement methodology,
|
||||
and known-flaky tests.
|
||||
- **0b (implementation):** all seven services (`gateway`, `ledger`, `fraud`,
|
||||
`sanctions`, `insights`, `notifier`, `audit`), `db/schema.sql` +
|
||||
`optional_indexes.sql`, `scripts/seed.py` + `smoke.py`, the frontend, and
|
||||
`docker-compose.yml`. Seeded: 800 accounts, 250,000 transactions, 45
|
||||
merchant categories, 12,000 sanctions entries, in ~12s. `make smoke` and
|
||||
`make test-contract` (64/64) pass; `make test-baseline` and
|
||||
`make test-pathology` pass except for a handful of tests documented below
|
||||
as load-sensitive rather than broken.
|
||||
- **Two changes to the frozen test contract, both made only after flagging
|
||||
them and getting an explicit go-ahead — not applied unilaterally:**
|
||||
- `test_row_counts_match_manifest_and_targets` loosened the `transactions`
|
||||
row-count check from exact equality to `>=`, since other contract tests
|
||||
in the same `make test-contract` run legitimately insert rows first.
|
||||
- `test_pool_starvation_plateaus_throughput` was **removed outright**.
|
||||
Its baseline assertion (throughput scales ≥2.5x from concurrency 5→25)
|
||||
is incompatible with this same spec's own §3 mandate of one uvicorn
|
||||
worker per service — confirmed by reproducing the identical
|
||||
non-scaling behaviour on a bare `GET /healthz` with zero backend
|
||||
logic. `pool_starvation` itself is still fully implemented
|
||||
(`services/ledger/db.py`'s dual-engine pool selection); only that one
|
||||
differential test was dropped.
|
||||
- **Known-flaky, documented rather than papered over:**
|
||||
`test_clean_baseline_is_healthy`, `test_nightmare_degrades_but_survives`,
|
||||
the baseline half of `test_blocking_event_loop_serializes_concurrent_requests`,
|
||||
and `test_missing_index_forces_seq_scan` compare a fast-path measurement
|
||||
against a slow-path one under real concurrent load, and pass at roughly a
|
||||
60–80% rate on ordinary dev hardware depending on momentary host
|
||||
contention. See "Known-flaky tests" in `tests/README.md`.
|
||||
- **Notable implementation deviations** (none change observable behaviour of
|
||||
the 13 pathologies or the request/response contract): gateway's `GET /`
|
||||
serves the frontend rather than §4's generic info JSON (the other five
|
||||
HTTP services still return that JSON as specified); `insights` splits a
|
||||
cheap PK-only `quick_summary` (embedded in every transaction response)
|
||||
from the real cached/N+1-affected 120-day analytics endpoint, so the
|
||||
missing-index pathology doesn't tax every request; `retry_storm` fires
|
||||
all 4 attempts unconditionally rather than stopping at first success
|
||||
(spec's "no circuit breaker" taken literally — also the only way to reach
|
||||
the required lookup-multiplication ratio given `sanctions_errors`' fixed
|
||||
3% rate); `poison_data` resolves via a Redis hash populated at seed time
|
||||
instead of a live table scan, to avoid the deliberately missing index;
|
||||
uvicorn's access log is disabled fleet-wide (redundant with the spec's
|
||||
own logging design, and was double-counting two log-line-based pathology
|
||||
tests); Postgres `max_connections` raised from 100 to 300 (five
|
||||
services' pool sizes can sum past the default under load).
|
||||
|
||||
**Time-box:** 2–3 evenings. Two agent sessions plus one evening of your own
|
||||
reading.
|
||||
|
||||
**Deliverable:** the working Ledgerline system with no observability, plus a
|
||||
black-box integration suite that pins down both its working state and its
|
||||
deliberate brokenness.
|
||||
|
||||
### Why TDD here, when the app itself doesn't matter
|
||||
|
||||
The microservices are throwaway. The *properties* aren't. Everything from Phase 2
|
||||
onward assumes the system still behaves a specific way: seven real services in
|
||||
the path, async work genuinely off the request path, five hot accounts owning a
|
||||
third of the rows, and thirteen pathologies that actually degrade something. You
|
||||
will be editing all seven services for weeks. Without a suite that fails when one
|
||||
of those properties quietly breaks, you'll spend an evening chasing a trace
|
||||
anomaly that turns out to be a regression you introduced yesterday.
|
||||
|
||||
Writing the tests first also forces the contract to be explicit before anything
|
||||
is built — which is what makes a second agent able to implement against it
|
||||
without guessing.
|
||||
|
||||
### 0a — the test suite
|
||||
|
||||
Agent session one, from `phase-0a-test-suite-spec.md`. That agent writes only
|
||||
`tests/` and must not create so much as a stub under `services/`. Its output
|
||||
should be **red**, with a `tests/RED-BASELINE.md` recording why each test fails
|
||||
right now — a test that passes against an empty repo is a broken test.
|
||||
|
||||
The suite asserts goals, not mechanics. Two representative examples worth
|
||||
understanding before you read the code:
|
||||
|
||||
- *"Stop the `sanctions` container and transactions must fail."* This is the test
|
||||
that catches an implementation quietly collapsing seven services into one
|
||||
process. Nothing else would.
|
||||
- *"With `n_plus_one` on, one insights request issues ≥ 50 SQL statements."*
|
||||
Measured via `pg_stat_statements`, not wall clock — because a query count
|
||||
can't be faked with a `sleep()`, and a latency threshold can.
|
||||
|
||||
That second point is the design principle throughout: **prefer assertions that
|
||||
are hard to satisfy without the real mechanism.** Query counts, `EXPLAIN` plan
|
||||
node types, Redis consumer-group lag, container RSS, and genuine unhandled
|
||||
exception paths all have that property.
|
||||
|
||||
### 0b — the implementation
|
||||
|
||||
Agent session two, from `phase-0-base-project-spec.md`, with `tests/` handed over
|
||||
as a **read-only contract**. That agent may not edit, skip, `xfail` or loosen a
|
||||
single test. If something looks impossible it stops and reports rather than
|
||||
adjusting the target.
|
||||
|
||||
Done when `make test` is green three consecutive runs and
|
||||
`git diff --stat tests/` is empty.
|
||||
|
||||
### Then, yourself
|
||||
|
||||
1. **Read every service's `app.py` end to end.** You need to know this codebase
|
||||
cold, because from Phase 2 onward you're modifying all seven services.
|
||||
2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
|
||||
then try to figure out what's wrong using only `docker compose logs`.** Give it
|
||||
twenty real minutes. You will fail, and the specific way you fail — no timing,
|
||||
no causality, no cross-service correlation, a `X-Request-Id` that dies at the
|
||||
Redis Stream — is the motivation for everything that follows. Write down what
|
||||
you wished you had. That list is your acceptance criteria for Phase 8.
|
||||
|
||||
**Done when:** `make test` is green three runs in a row, all nine acceptance
|
||||
criteria in the base spec pass, and you have that written list of frustrations.
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Phase 1 — The model, and one signal end to end
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Blocked on nothing — Phase 0 is done and the platform is
|
||||
described above. Waiting to begin.
|
||||
|
||||
**Time-box:** one evening.
|
||||
|
||||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||||
than abstract.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||||
Observability is a property of the system: can you ask a *new* question about
|
||||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||||
attributes), not just pre-aggregated counters.
|
||||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||||
- The **API** is what your code and third-party instrumentation libraries call.
|
||||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||||
library can be instrumented without forcing a dependency on any implementation.
|
||||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||||
Application owners configure the SDK; library authors only touch the API.
|
||||
- The **Collector** is a standalone process that receives, processes and exports
|
||||
telemetry. Optional, but you should use one.
|
||||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||||
that process. Attributes describe the individual span/metric/log.
|
||||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||||
aware of, not learning yet.)
|
||||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||||
seven services. The Collector also does the things you don't want in your app:
|
||||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||||
destinations.
|
||||
|
||||
### Resources
|
||||
|
||||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||||
it — just see what "done" looks like.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||||
`ansible-galaxy collection install -r requirements.yml`, change
|
||||
`grafana_admin_password` (vault it), then
|
||||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||||
a single container with everything pre-wired. Useful for a first trace, but
|
||||
don't build on it — its pipeline differs from your real platform in exactly the
|
||||
ways that matter.)*
|
||||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||||
auto-instrumentation:
|
||||
```
|
||||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||||
opentelemetry-bootstrap -a install
|
||||
```
|
||||
and change its command to
|
||||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||||
with `OTEL_SERVICE_NAME=sanctions`,
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||||
Tempo directly.
|
||||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||||
|
||||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||||
loud what the Resource attributes on it mean and where they came from.
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Phase 10 — Package the delivery
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on every prior phase being done — this is the wrap-up,
|
||||
not independent work.
|
||||
|
||||
**Time-box:** one evening.
|
||||
|
||||
Babak gave you a task; deliver it like an engineer, not like a student.
|
||||
|
||||
1. **`README.md`** — architecture diagram, one-command start, what each container is
|
||||
for, and a "here's the demo" script.
|
||||
2. **The five-minute demo path**, rehearsed:
|
||||
- Delete every dashboard in the Grafana UI, then `ansible-playbook
|
||||
playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
|
||||
from code, one command.
|
||||
- Submit one transaction from the frontend, then follow it through Grafana: RED
|
||||
dashboard → latency spike → exemplar → trace → span → logs for that span → back
|
||||
to the trace. One request, all three signals, no searching.
|
||||
- `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
|
||||
and the service map turn red at a specific node.
|
||||
- Show `docs/INVESTIGATION.md` and how you found the pathologies.
|
||||
3. **The written answers** Babak asked for, as documents in the repo, not as a
|
||||
conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
|
||||
`docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
|
||||
RTO/RPO/SLA/SLO/error budget in your own words.
|
||||
4. **The test suite as part of the story.** Show `make test` green, then show the
|
||||
three generations of the call-counting assertion side by side —
|
||||
`docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
|
||||
each signal bought you. Most people demo dashboards; almost nobody can show
|
||||
their observability work paying off inside their own test suite.
|
||||
5. **A "what I'd do differently in production" section.** Tail sampling instead of
|
||||
100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
|
||||
log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
|
||||
you *didn't* do, and why, is the strongest signal of seniority in the whole
|
||||
delivery.
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Phase 2 — Distributed tracing
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
|
||||
|
||||
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
|
||||
|
||||
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
|
||||
(Resource, providers, exporters, Collector wiring) is shared across all three
|
||||
signals — do it once here and logs and metrics become incremental. Second, tracing
|
||||
is what Babak actually asked for ("trace a single request, see where it stuck"),
|
||||
and once trace IDs exist, log correlation in Phase 3 works on the first try instead
|
||||
of needing rework.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Trace / span / span context.** A trace is one request's journey; a span is one
|
||||
timed operation within it. Each span carries a trace ID, its own span ID, and its
|
||||
parent's span ID. That parent pointer is the entire data structure.
|
||||
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
|
||||
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
|
||||
these right is what makes the service graph render correctly.
|
||||
- **Context propagation.**
|
||||
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
|
||||
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
|
||||
carry the context explicitly.
|
||||
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
|
||||
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
|
||||
- The classic bug: any boundary that doesn't carry those headers **starts a brand
|
||||
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
|
||||
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
|
||||
producer and `.extract(...)` on the consumer, then starting the consumer span
|
||||
with that context as parent. This is the single most valuable exercise in the
|
||||
phase — most people never do it manually and never really understand it.
|
||||
- Related: `links` (for a consumer span that batches messages from many traces)
|
||||
and **Baggage** (arbitrary key/values propagated alongside trace context —
|
||||
useful, and a foot-gun, because it's sent on every hop and is visible to
|
||||
downstream systems; never put secrets in it).
|
||||
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
|
||||
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
|
||||
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
|
||||
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
|
||||
that show *which service* was slow but not *why*.
|
||||
- **Span attributes vs metric labels.** Spans are the right home for
|
||||
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
|
||||
with volume, not with distinct values. This is why "put it on the span, not the
|
||||
metric" is the standing rule.
|
||||
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
|
||||
That's how the exception ends up attached to the trace and how the `spanmetrics`
|
||||
connector in Phase 4 will count your error rate.
|
||||
- **Sampling — the concept now, the config later.**
|
||||
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
|
||||
errors at the same rate as successes.
|
||||
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
|
||||
traces, but must run in the Collector and buffer spans in memory.
|
||||
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
|
||||
now; you'll configure tail sampling in the Collector in Phase 5.
|
||||
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
|
||||
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
|
||||
unilaterally drop half of a trace its caller decided to keep.
|
||||
|
||||
### Resources
|
||||
|
||||
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
|
||||
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
|
||||
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
|
||||
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
|
||||
practical FastAPI-specific setup, including batch/queue env vars.
|
||||
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
|
||||
reading.
|
||||
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
|
||||
side by side. Look specifically at how they mix auto-instrumentation with
|
||||
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
|
||||
This is the "professional implementation" you asked to study.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
|
||||
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
|
||||
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
|
||||
service calls it once at startup. Keep it under 60 lines.
|
||||
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
|
||||
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
|
||||
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
|
||||
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
|
||||
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
|
||||
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
|
||||
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
|
||||
`ledgerline.tx.amount_minor`).
|
||||
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
|
||||
inject trace context into the message fields. Consumers (`notifier`, `audit`):
|
||||
extract, and start a `CONSUMER` span with the extracted context as parent.
|
||||
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
|
||||
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
|
||||
more.
|
||||
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
|
||||
the "before" state so you could measure against it, and from here it's
|
||||
actively wrong. Everything else in `tests/` must still pass: instrumenting
|
||||
the system is not allowed to change its behaviour, and if `make test` goes red
|
||||
after adding spans, you've broken something real. That is exactly the safety
|
||||
net you built it for.
|
||||
|
||||
### Done when
|
||||
|
||||
- One request produces **one connected trace** covering all seven services,
|
||||
including both async consumers.
|
||||
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
|
||||
ate the time.
|
||||
- You can explain why `blocking_event_loop` looks different in a trace than
|
||||
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
|
||||
before spans start, plus every concurrent request in that process degrading
|
||||
together.)
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Phase 3 — Structured logs, correlated to traces
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 2 (trace IDs must exist before log correlation
|
||||
means anything).
|
||||
|
||||
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
|
||||
events: a stable event name plus typed fields (`event="transaction_settled"`,
|
||||
`account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
|
||||
Consistent field names across all seven services — `account_id` everywhere, never
|
||||
`accountId` in one place and `acct` in another. This is what makes LogQL queries
|
||||
possible at all.
|
||||
- **Trace–log correlation.** Every log line emitted inside an active span must carry
|
||||
`trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
|
||||
"one pane of glass": click a span → see its logs; click a log line → open its
|
||||
trace. Both directions are configured, and you'll set both up.
|
||||
- **Two ways to ship logs, and which one your platform actually does:**
|
||||
- **stdout → collector scraping → Loki.** The container-native way. Your app just
|
||||
prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
|
||||
standard in Kubernetes. **This is the only path wired in your Alloy config**
|
||||
(`loki.source.docker` reading the Docker socket).
|
||||
- **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
|
||||
with the same Resource attributes attached automatically. Cleaner correlation,
|
||||
but the app owns delivery. **On your platform this currently goes nowhere** —
|
||||
Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
|
||||
dropped in silence.
|
||||
- Do both anyway: use stdout for most services, and *extend Alloy* to wire the
|
||||
OTLP logs path for one or two. Then argue the trade-off from experience. That
|
||||
comparison — including the fact that a misconfigured pipeline fails silently
|
||||
rather than loudly — is precisely what Babak will probe.
|
||||
- **The field name is not a style choice here.** Your Loki datasource is provisioned
|
||||
with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
|
||||
A log line must carry lowercase `trace_id`. OpenTelemetry's logging
|
||||
auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
|
||||
logs and traces both arriving, no errors anywhere, and no clickable link. Hence
|
||||
structlog with an explicit processor rather than the auto-injection shortcut.
|
||||
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
|
||||
your entire fleet, and Loki bills/indexes on volume.
|
||||
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
|
||||
`json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
|
||||
redaction into the processor chain, not into each call site.
|
||||
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
|
||||
not log content. Labels should be few and low-cardinality (`service`, `env`,
|
||||
`level`). Everything else lives in the log line body and is filtered at query time.
|
||||
Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
|
||||
in the JSON body and use a **derived field** to make it clickable.
|
||||
|
||||
### Resources
|
||||
|
||||
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
|
||||
Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
|
||||
setup, including the hex formatting for trace/span IDs.
|
||||
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
|
||||
`OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
|
||||
fields, if you'd rather not hand-roll a processor.
|
||||
- Udemy backbone: the Loki + LogQL sections.
|
||||
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
|
||||
|
||||
### Build
|
||||
|
||||
1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
|
||||
processor that pulls the current span context and injects `trace_id` / `span_id`.
|
||||
2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
|
||||
handler: request received (or rely on the span), business decision, error.
|
||||
3. Add a redaction processor with a denylist of field names.
|
||||
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
|
||||
with a `container` label (Alloy's relabel rule sets it from the Docker container
|
||||
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
|
||||
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
|
||||
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
|
||||
the receiver shows traffic, and the pipeline shows no logs output. Now extend
|
||||
`config.alloy.j2` — add a `logs` output on the receiver plus an
|
||||
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
|
||||
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
|
||||
That is the most transferable half-hour in the whole project.
|
||||
6. Confirm the provisioned correlation works in both directions: Loki's
|
||||
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
|
||||
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
|
||||
are verifying, not writing. If a link doesn't appear, the regex didn't match
|
||||
your field name — check that before anything else.
|
||||
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
|
||||
(`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
|
||||
in `docker compose logs` — the only cross-service call counter that existed at
|
||||
Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
|
||||
with LogQL over structured fields instead. This is a small change and a
|
||||
pointed one: notice how much shorter and more precise the assertion gets, and
|
||||
that it now works without shelling out to Docker at all. That delta is the
|
||||
value of structured logging, measured on your own code.
|
||||
|
||||
**Done when:** from a slow trace you can click into that request's logs, and from
|
||||
any log line you can click into its trace — in both directions, in two clicks.
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
# Phase 4 — Metrics
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Not blocked technically (metrics don't strictly need tracing
|
||||
first), but sequenced after Phase 3 per the roadmap's ordering.
|
||||
|
||||
**Time-box:** 2 evenings.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Instrument types.**
|
||||
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
|
||||
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
|
||||
connection-pool checkouts.
|
||||
- **Histogram** — a distribution. Request latency, payload size. This is what
|
||||
gives you percentiles.
|
||||
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
|
||||
read via a callback on collection (right for things you *poll*, like pool size).
|
||||
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
|
||||
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
|
||||
nonsense. Use histograms.
|
||||
- **Cardinality — the one thing that will bite you.** Every unique combination of
|
||||
metric name + label values is a separate time series in memory. Add `account_id`
|
||||
as a label on a 800-account demo and you've made 800 series; do it in production
|
||||
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
|
||||
~10 000 distinct values in production, it does not belong on a metric.** Put it on
|
||||
a span. Histograms multiply this (one series per bucket per label combination), so
|
||||
don't over-bucket either.
|
||||
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
|
||||
the concrete path), `http.status_code` class, `kind`, `currency`.
|
||||
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
|
||||
raw URL, timestamp, error message.
|
||||
- **RED and USE.**
|
||||
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
|
||||
per-service dashboard.
|
||||
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
|
||||
connection pools, queues, disk. Saturation is the one everyone forgets and the
|
||||
one that predicts collapse.
|
||||
- You want both: RED tells you the service is slow, USE tells you which resource
|
||||
is why.
|
||||
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
|
||||
and knowing both is the point:
|
||||
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
|
||||
`servicegraph` connectors read spans in flight and emit RED metrics and edge
|
||||
metrics. This is what most blog posts (including the OneUptime article) show.
|
||||
- *In the trace store:* Tempo's **metrics-generator** does the same job after
|
||||
ingestion, remote-writing the result to Prometheus. **This is what your platform
|
||||
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
|
||||
`send_exemplars: true`, remote-writing to a Prometheus started with
|
||||
`--web.enable-remote-write-receiver`.
|
||||
- So RED metrics, the service map and exemplars **already exist** before you write
|
||||
a single metric line. Do not add the collector connectors as well — you'd
|
||||
duplicate every series under a second set of names. **Go find the real metric
|
||||
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
|
||||
they will not match the article's.
|
||||
- The trade-off worth being able to argue: collector-side generation is cheaper
|
||||
and works before storage, but sees only what passes through that collector;
|
||||
Tempo-side generation sees everything that landed, survives collector restarts,
|
||||
and can be enabled per-tenant — at the cost of load on the trace store.
|
||||
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
|
||||
click a spike on a latency graph and land in the exact trace. On your platform
|
||||
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
|
||||
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
|
||||
`name: trace_id`. If you emit your own histograms with exemplars, the label must
|
||||
be `trace_id` to match.
|
||||
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
|
||||
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
|
||||
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
|
||||
the pull model is why Prometheus has service discovery and why `up` exists.
|
||||
|
||||
### Resources
|
||||
|
||||
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
|
||||
Read the first half. Best cardinality explanation available.
|
||||
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
|
||||
not a fact.
|
||||
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
|
||||
`servicegraph` connector config and its RED PromQL queries almost verbatim.
|
||||
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
|
||||
|
||||
### Build
|
||||
|
||||
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
|
||||
2. Custom application metrics, deliberately few:
|
||||
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
|
||||
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
|
||||
- `ledgerline.fraud.score` (Histogram; no labels)
|
||||
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
|
||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
||||
metrics, so pick a path and implement it:
|
||||
- *Platform grain:* expose `/metrics` on each service and add them to
|
||||
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
|
||||
playbook. Prometheus scrapes your services directly. Note the irony and the
|
||||
lesson — you're back to a pull model and a second client library, which is
|
||||
exactly the fragmentation OTel exists to fix.
|
||||
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
|
||||
Alloy so app metrics flow through the collector as the architecture drawing
|
||||
intends. This also closes a real diagram-vs-code gap in the repo.
|
||||
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
|
||||
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
|
||||
all three signals.
|
||||
Do the first to get moving; do one of the others afterwards and write up which
|
||||
you'd choose for a real platform team and why. That write-up is a Phase 9
|
||||
artefact.
|
||||
4. Deliberately break it: add `account_id` as a label to
|
||||
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
|
||||
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
|
||||
you'll never forget.
|
||||
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
|
||||
one. Understand why they differ slightly and when you'd want each.
|
||||
6. **Second migration of the call-counting assertions.** The three tests you moved
|
||||
to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
|
||||
TraceQL query for `sanctions` client spans within a single trace answers
|
||||
"how many lookups did this one transaction make?" directly, which is the
|
||||
question the test was always really asking. Do it and compare all three
|
||||
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
|
||||
clearest possible demonstration of what each signal is actually for, on a
|
||||
problem you already understand.
|
||||
|
||||
**Done when:** you have a RED view per service, a working service map, and you can
|
||||
click a latency spike through to a trace via an exemplar.
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# Phase 5 — Dashboards and alerts as code
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
|
||||
dashboarding.
|
||||
|
||||
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
|
||||
recreate in seconds.*
|
||||
|
||||
### Concepts
|
||||
|
||||
- Three levels of "as code", and you should know when each applies:
|
||||
1. **File provisioning** — YAML in `provisioning/datasources/` and
|
||||
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
|
||||
container. Grafana loads them at startup. Simplest, no state, no API tokens,
|
||||
perfect for a Compose demo. **This alone satisfies the requirement.**
|
||||
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
|
||||
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
|
||||
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
|
||||
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
|
||||
of Grafonnet (which is legacy and unofficial).
|
||||
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
|
||||
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
|
||||
when this lands on your Kubernetes cluster.
|
||||
- **Dashboards should answer questions, not display data.** A good service dashboard
|
||||
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
|
||||
is the pressure. Panels that nobody looks at during an incident are cost, not value.
|
||||
- **Alerts as code too.** Grafana alert rules, contact points and notification
|
||||
policies all provision from YAML. And keep the *alert definition* next to the
|
||||
dashboard that explains it.
|
||||
|
||||
### Resources
|
||||
|
||||
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
|
||||
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
|
||||
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
|
||||
example.
|
||||
- Udemy backbone: dashboard design sections (panel types, transformations,
|
||||
variables, dashboard best practice).
|
||||
|
||||
### Build
|
||||
|
||||
Your platform provisions **datasources only** — there is no dashboards provider at
|
||||
all. So this phase is entirely yours, and the right home for it is inside the
|
||||
Ansible role, using the same mechanism as everything else.
|
||||
|
||||
1. **Extend the role.** Add
|
||||
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
|
||||
(a file-based dashboard provider pointing at a mounted directory), a
|
||||
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
|
||||
and copy them. The Compose template already mounts
|
||||
`./config/grafana/provisioning` read-only, so the provider is picked up on
|
||||
restart.
|
||||
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
|
||||
Foundation SDK (Python)**: one function producing a RED dashboard for a given
|
||||
service name, looped over the seven services; a second producing a system
|
||||
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
|
||||
the generator *and* its output; the Ansible role ships the output.
|
||||
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
|
||||
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
|
||||
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
|
||||
than a local Makefile target, because it's the same command that builds the
|
||||
platform.
|
||||
4. While you're in there, review the provisioned datasources and add what's missing:
|
||||
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
|
||||
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
|
||||
understand why `filterByTraceID: true` alone is sufficient here, and when it
|
||||
wouldn't be.
|
||||
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
|
||||
|
||||
**Done when:** re-running the playbook restores every dashboard from scratch, and no
|
||||
dashboard has ever been saved through the Grafana UI.
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# Phase 6 — Load testing with k6, and percentiles
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
|
||||
benefits from Phase 5's dashboards existing.
|
||||
|
||||
**Time-box:** 1–2 evenings.
|
||||
|
||||
### Concepts — P95 and P99 properly
|
||||
|
||||
You asked about these specifically, and there's more to them than "the 95th
|
||||
percentile."
|
||||
|
||||
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
|
||||
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
|
||||
is.
|
||||
- **You cannot average percentiles.** Not across instances, not across time buckets.
|
||||
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
|
||||
to aggregate the *histogram buckets* and compute the quantile from the sum:
|
||||
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
|
||||
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
|
||||
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
|
||||
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
|
||||
bracket your actual SLO threshold. (Native histograms largely solve this and are
|
||||
worth knowing about.)
|
||||
- **Which population?** An overall p99 across all endpoints is dominated by whichever
|
||||
endpoint has the most traffic. Slice by route, or you'll miss that
|
||||
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
|
||||
- **Tail latency amplification** — the most important idea here for a microservice
|
||||
system. If one user request fans out to 10 backend calls, each with an independent
|
||||
p99 of 1 s, then the chance the *user request* touches at least one slow call is
|
||||
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
|
||||
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
|
||||
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
|
||||
measurable — go measure it.
|
||||
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
|
||||
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
|
||||
that.
|
||||
- **Coordinated omission** — a load-generator trap. If your generator waits for a
|
||||
response before sending the next request, then when the system stalls, the
|
||||
generator stalls too and simply *doesn't record* the requests that would have been
|
||||
slow. Your measured latency is systematically optimistic. Fix: use an **open
|
||||
model** — a fixed *arrival rate* independent of response times. In k6 that means
|
||||
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
|
||||
`constant-vus`. Knowing this is a genuine senior-level differentiator.
|
||||
|
||||
### Concepts — k6
|
||||
|
||||
- **Test types:** smoke (does it work at all), average-load (normal traffic),
|
||||
stress (past normal, find the knee), spike (sudden surge, test recovery),
|
||||
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
|
||||
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
|
||||
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
|
||||
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
|
||||
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
|
||||
fail the run; **thresholds are SLOs expressed in code and they set the process exit
|
||||
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
|
||||
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
|
||||
view and the application's view sit on the same Grafana time axis. That side-by-side
|
||||
is the single most convincing screenshot you can show Babak.
|
||||
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
|
||||
writing 60 lines of it.
|
||||
|
||||
### Resources
|
||||
|
||||
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
|
||||
*Thresholds*. Free, genuinely well written; ~2 h to work through.
|
||||
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
|
||||
hands-on).
|
||||
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
|
||||
- Grafana docs → *k6 output to Prometheus remote write*.
|
||||
|
||||
### Build
|
||||
|
||||
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
|
||||
sharing a `common.js` for setup and payload generation against random accounts.
|
||||
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
|
||||
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
|
||||
`http_req_failed rate<0.01`.
|
||||
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
|
||||
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
|
||||
sets. This before/after pair is your baseline for Phase 8.
|
||||
6. Optional but impressive: have k6 send W3C trace context headers so load-test
|
||||
requests appear as traces, letting you click from a k6 spike into a specific
|
||||
trace. Check the current k6 docs for the tracing module's status — it has moved
|
||||
between experimental and stable.
|
||||
|
||||
**Done when:** `make load-stress` produces a Grafana view where you can see the
|
||||
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
|
||||
service that broke first.
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 6's load-test baseline (the latency SLO should
|
||||
come from measured numbers, not be picked from thin air).
|
||||
|
||||
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
|
||||
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
|
||||
`POST /api/transactions` requests that returned 2xx within 500 ms.*
|
||||
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
|
||||
requests succeed within 500 ms, measured over a rolling 28 days."
|
||||
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
|
||||
Always **looser** than your SLO, so you have room to miss internally without
|
||||
breaching commercially. If your SLA and SLO are the same number, you've made a
|
||||
mistake.
|
||||
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
|
||||
budget; over 30 days that's about **43 minutes**. The budget is the thing that
|
||||
turns reliability from an argument into arithmetic: if there's budget left, ship
|
||||
features and take risks; if it's spent, the policy says you stop feature work and
|
||||
spend on reliability. That policy — agreed in advance by dev, ops and product — is
|
||||
the whole point.
|
||||
- **Burn rate.** How fast you're consuming budget relative to the window.
|
||||
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
|
||||
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
|
||||
is what stops you from either paging on every blip or noticing the slow bleed a
|
||||
month late.
|
||||
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
|
||||
and it's worth being precise because interviewers probe exactly here.
|
||||
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
|
||||
from failure to restored service. It drives your architecture: replication,
|
||||
warm standby, failover automation, runbook quality.
|
||||
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
|
||||
in time: how far back can you afford to lose. It drives your **backup and
|
||||
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
|
||||
every 15 minutes.
|
||||
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
|
||||
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
|
||||
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
|
||||
5 minutes of transactions." Both feed the SLA.
|
||||
- Both are **business decisions, not technical ones** — you derive them by asking
|
||||
what an hour of downtime or an hour of lost data actually costs, then buying the
|
||||
cheapest architecture that meets it. Tightening either is expensive and
|
||||
non-linear.
|
||||
|
||||
### Resources
|
||||
|
||||
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
|
||||
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
|
||||
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
|
||||
all free PDFs. Run the exercises solo with Ledgerline as the subject.
|
||||
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
|
||||
free, self-paced, deeper.
|
||||
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
|
||||
overview; pair it with your Postgres backup knowledge.
|
||||
|
||||
### Build
|
||||
|
||||
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
|
||||
- Availability SLI/SLO on `POST /api/transactions`
|
||||
- Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
|
||||
from thin air)
|
||||
- A freshness/lag SLI on the notifier consumer group
|
||||
- A stated SLA that is deliberately looser than each SLO
|
||||
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
|
||||
- An error budget policy: what happens at 50%, 75% and 100% consumed
|
||||
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
|
||||
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
|
||||
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
|
||||
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
|
||||
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
|
||||
be able to argue the alternative: Grafana unified alerting can evaluate across
|
||||
datasources (including Loki queries) and lives with your dashboards, whereas
|
||||
Alertmanager is metrics-only but is the ecosystem standard and gives you
|
||||
grouping, deduplication, inhibition and silences for free. Note which you chose
|
||||
and why in `docs/SLO.md`.
|
||||
3. Add an error-budget panel to the overview dashboard: budget remaining, current
|
||||
burn rate, projected exhaustion.
|
||||
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
|
||||
watch the fast-burn alert fire.
|
||||
|
||||
**Done when:** you can explain, in one paragraph each and without notes, the
|
||||
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
|
||||
decision.
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# Phase 8 — Bottleneck hunting (the capstone)
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
|
||||
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
|
||||
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
|
||||
|
||||
**Time-box:** 2 evenings. This is where the whole thing pays off.
|
||||
|
||||
### Concepts — how to actually answer "how do I find bottlenecks?"
|
||||
|
||||
Babak asked how you'd *think* about this, not for a list of tools. Here's the
|
||||
method, roughly in order. The discipline is to go **top-down from the user, not
|
||||
bottom-up from the infrastructure** — bottom-up is how people spend three hours
|
||||
tuning a database that wasn't the problem.
|
||||
|
||||
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
|
||||
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
|
||||
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
|
||||
2. **Localise to a service** using RED + the service graph. Which node's duration
|
||||
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
|
||||
the graph from the leaves inward.
|
||||
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
|
||||
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
|
||||
several slow traces plus one fast trace of the same endpoint as a control.
|
||||
4. **Read the trace for five specific shapes:**
|
||||
- **A long span** — the work itself is slow. Look at its attributes: which query,
|
||||
which host, which parameters.
|
||||
- **A gap** — wall-clock time not covered by any span. This is the interesting
|
||||
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
|
||||
blocked event loop, or a missing instrumentation library. Ledgerline's
|
||||
`pool_starvation` and `blocking_event_loop` both show up as gaps.
|
||||
- **Serialisation** — spans laid end to end that could have been concurrent.
|
||||
Ledgerline's `no_batching` looks exactly like this.
|
||||
- **Repetition** — the same span name 150 times in a row. That's your N+1.
|
||||
- **Retries** — the same client span repeated 3–4 times with errors. That's
|
||||
`retry_storm`, and note that it makes an *overload* worse, never better.
|
||||
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
|
||||
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
|
||||
memory. A resource at 100% utilisation with a growing queue is saturated;
|
||||
latency there is a *symptom*, not the cause.
|
||||
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
|
||||
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
|
||||
concurrent slots. If your pool is 5, you queue — and queueing latency grows
|
||||
non-linearly as you approach saturation. This one formula explains most
|
||||
connection-pool and worker-count incidents you'll ever see.
|
||||
7. **Check for tail amplification** (Phase 6) before blaming any single service.
|
||||
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
|
||||
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
|
||||
over the *same duration*. One variable at a time. Never declare a fix from a
|
||||
dashboard that "looks better."
|
||||
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
|
||||
becomes the bottleneck. Re-measure after every change; stop when you're inside
|
||||
the SLO, not when you run out of ideas.
|
||||
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
|
||||
or lock waits inside Postgres, or which line of Python burned the cycles. That's
|
||||
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
|
||||
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
|
||||
|
||||
### Resources
|
||||
|
||||
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
|
||||
for saturation.
|
||||
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
|
||||
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
|
||||
thing ever written on why microservice latency behaves the way it does.
|
||||
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
|
||||
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
|
||||
these; connect them to the trace's DB spans.
|
||||
|
||||
### Build
|
||||
|
||||
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
|
||||
`docs/PATHOLOGIES.md`.**
|
||||
2. Run `make load-average` for 10 minutes.
|
||||
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
|
||||
like a real incident doc: what you observed, what you hypothesised, what you
|
||||
checked, what it turned out to be. Include screenshots.
|
||||
4. When you think you've found them all, run `make chaos-clean`, enable them one at
|
||||
a time, and confirm each signature.
|
||||
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
|
||||
6. Repeat once with a new random set. The second run is where the method actually
|
||||
becomes yours.
|
||||
|
||||
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
|
||||
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Not technically blocked by earlier phases (it stands on its own
|
||||
as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
|
||||
re-diagnose in Kibana already exists.
|
||||
|
||||
**Time-box:** 1–2 evenings.
|
||||
|
||||
Babak's remaining questions are all *judgment* questions. He isn't testing whether
|
||||
you can recite what Filebeat does; he's testing whether you have a framework for
|
||||
choosing. Here's the framework, then the answers as worked examples.
|
||||
|
||||
### The framework — six axes to reason on
|
||||
|
||||
Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
|
||||
|
||||
1. **Where does the data originate?** Inside your application (you can instrument it)
|
||||
or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
|
||||
touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
|
||||
2. **What shape is it already in?** Structured JSON at the source, or unstructured
|
||||
text needing grok/regex parsing? Parsing cost has to land *somewhere*.
|
||||
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
|
||||
4. **What's the destination, and how many?** One backend forever, or several, or one
|
||||
you expect to change?
|
||||
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
|
||||
on every host) distributes cost and adds resilience but multiplies config surface.
|
||||
Parsing centrally concentrates cost and creates a single point of failure — but
|
||||
one place to change rules.
|
||||
6. **What already exists, and what does it cost?** Existing Elastic investment,
|
||||
existing licences, team skills, and licensing constraints (Elastic's Filebeat and
|
||||
Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
|
||||
Bit and Vector are permissively licensed). Migration is never free.
|
||||
|
||||
### The answers
|
||||
|
||||
**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
|
||||
from a host or container and forwards them with minimal transformation — and your
|
||||
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
|
||||
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
|
||||
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
|
||||
included. Reach for it when the source is a file you don't control the format of, the
|
||||
app can't be modified, and Elasticsearch is where it's going.
|
||||
|
||||
**When to use Logstash.** When you need real *transformation*: parsing unstructured
|
||||
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
|
||||
splitting or merging events, conditional routing to multiple destinations, or
|
||||
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
|
||||
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
|
||||
per host.
|
||||
|
||||
**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
|
||||
architecture and the reason is a **division of labour between edge and centre**:
|
||||
|
||||
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
|
||||
backpressure, ships onward. You want the thing running in a thousand places to be
|
||||
small and boring.
|
||||
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
|
||||
enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
|
||||
you want them in a handful of places you can redeploy, not on a thousand hosts.
|
||||
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
|
||||
persistent queue absorbs it and Filebeat backs off gracefully, rather than every
|
||||
host queuing independently or dropping.
|
||||
|
||||
So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
|
||||
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
|
||||
need central buffering. Skip Logstash when your logs are already structured JSON and
|
||||
Elasticsearch ingest pipelines can do the little parsing that remains.
|
||||
|
||||
**When to use OpenTelemetry.**
|
||||
|
||||
- You own the application code and want telemetry generated *from inside it* — spans
|
||||
with business context, custom metrics, logs already carrying trace IDs.
|
||||
- You need **more than one signal**, correlated. This is the strongest argument: one
|
||||
SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
|
||||
threaded through all three. Neither Filebeat nor Logstash does traces.
|
||||
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
|
||||
data model across all of them.
|
||||
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
|
||||
Jaeger by editing Collector YAML.
|
||||
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
|
||||
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
|
||||
rewriting — without touching the app.
|
||||
|
||||
**When *not* to use OpenTelemetry.** This is the question that separates people who
|
||||
have used it from people who have read about it.
|
||||
|
||||
- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
|
||||
an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
|
||||
file-based collector.
|
||||
- **You only need one signal from one source with no correlation.** A single nginx
|
||||
access log going to a single log store: OTel is more machinery than the job needs.
|
||||
Fluent Bit or Filebeat is simpler and finishes the job today.
|
||||
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
|
||||
Elastic estate with hundreds of Beats modules, dashboards and detection rules is
|
||||
years of investment. "We should use OTel because it's the standard" is not a
|
||||
business case. Note the direction of travel though: the Collector can already
|
||||
receive from and export to most of these, so a *gradual* migration is usually the
|
||||
right shape — often by putting a Collector or Vector in front of the existing
|
||||
agents rather than ripping them out.
|
||||
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
|
||||
where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
|
||||
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
|
||||
a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
|
||||
retention, different query patterns, different cost model.
|
||||
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
|
||||
matured at different rates per language, and profiling is still alpha. Check the
|
||||
status page for your language before betting on a signal.
|
||||
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
|
||||
on it. A two-person team that needs error tracking by Friday should install Sentry.
|
||||
|
||||
The senior framing to give Babak: *these tools aren't competitors so much as
|
||||
different layers.* OpenTelemetry is a **generation and transport standard** for
|
||||
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
|
||||
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
|
||||
Elasticsearch are **stores**. Most real architectures use one from each layer, and
|
||||
the interesting question is always which layer a given problem lives in.
|
||||
|
||||
### Resources
|
||||
|
||||
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
|
||||
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
|
||||
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
|
||||
read these to see how the Collector does the Filebeat/Logstash job itself.
|
||||
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
|
||||
caveats.
|
||||
|
||||
### Build — and this is no longer hypothetical
|
||||
|
||||
Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
|
||||
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
|
||||
Filebeat reading Docker container logs and Logstash parsing JSON into daily
|
||||
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
|
||||
questions — the codebase he handed you contains both worlds.
|
||||
|
||||
1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
|
||||
flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
|
||||
Alloy → Loki. Compare them on the same data.
|
||||
2. Test the direct ingest path the README documents — pipe a JSON line into
|
||||
Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
|
||||
parsing happens before you change it.
|
||||
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
|
||||
that parses them. Then do the same job with Alloy's `loki.process` stages. Time
|
||||
both. Note which one you'd rather debug at 2 a.m.
|
||||
4. Compare the *investigation experience* directly: take one incident you already
|
||||
diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
|
||||
you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
|
||||
wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
|
||||
for some questions, and Loki's cost model genuinely beats Elasticsearch's for
|
||||
others.
|
||||
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
|
||||
licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
|
||||
permissively licensed. For some employers that alone decides it.
|
||||
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
|
||||
valuably — what you actually found annoying about each. Having run both stacks
|
||||
against the same application is the difference between reciting a comparison and
|
||||
arguing one.
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# Roadmap summary — all phases at a glance
|
||||
|
||||
*Part of `docs/roadmap/` — the one-page overview. Read [`intro.md`](intro.md)
|
||||
first for platform/background context; read the individual `phase-N.md` file
|
||||
for the phase you're actually working on. This file exists so an agent (or
|
||||
you) can check overall status and sequencing without loading every phase's
|
||||
full detail.*
|
||||
|
||||
Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase
|
||||
file follows the same shape: **Status**, **Message** (what's done, notes,
|
||||
warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
|
||||
## Status and focus
|
||||
|
||||
| Phase | File | Status | Focus | Time-box | Main output |
|
||||
|---|---|---|---|---|---|
|
||||
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
|
||||
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
|
||||
| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
|
||||
| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
||||
| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
||||
| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
|
||||
| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
|
||||
| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
|
||||
| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
|
||||
| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
|
||||
| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
|
||||
| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
|
||||
|
||||
## What each phase is, in one line
|
||||
|
||||
- **0 — Build the base project, test-first.** Write the black-box test suite,
|
||||
then implement the seven-service system against it as a frozen contract.
|
||||
**Done** — see `phase-0.md` for the full implementation report.
|
||||
- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
|
||||
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
|
||||
Ansible and get one auto-instrumented trace into Tempo.
|
||||
- **2 — Distributed tracing.** The heart of the task: auto- and
|
||||
manual-instrument all seven services, and hand-propagate trace context
|
||||
across the Redis Stream hop, which carries none by default.
|
||||
- **3 — Structured logs, correlated to traces.** Replace prose logging with
|
||||
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
|
||||
in both directions.
|
||||
- **4 — Metrics.** Instrument RED metrics per service, understand cardinality
|
||||
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
|
||||
silently. Tempo's metrics-generator already produces span metrics and the
|
||||
service graph — don't duplicate it in the collector.
|
||||
- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
||||
Grafana Foundation SDK and ship it through the Ansible role, so "destroy
|
||||
and recreate in seconds" is a real, repeatable command.
|
||||
- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't
|
||||
be averaged, what coordinated omission is, and drive Ledgerline with
|
||||
arrival-rate executors to get a clean before/after load profile.
|
||||
- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
|
||||
SLOs from the Phase 6 baseline (not from thin air), write an error-budget
|
||||
policy, and wire multi-burn-rate alerts.
|
||||
- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
|
||||
of chaos flags, find the pathologies from telemetry alone using a top-down,
|
||||
hypothesis-driven method, and write it up as an incident doc.
|
||||
- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
|
||||
OpenTelemetry on six concrete axes, using the platform's real second stack
|
||||
(Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
|
||||
- **10 — Package the delivery.** README, a rehearsed five-minute demo, the
|
||||
written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`,
|
||||
`CONCEPTS.md`), and an honest "what I'd do differently in production"
|
||||
section.
|
||||
|
||||
## If you have to cut scope
|
||||
|
||||
Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not
|
||||
cut Phase 8** — it's the only phase that proves you learned anything.
|
||||
Reference in New Issue
Block a user