LedgerLine/docs/roadmap/intro.md

# Observability & APM — Delivery Roadmap for Babak's Task

*Part of `docs/roadmap/` — platform background and reading guidance shared by
every phase. Per-phase detail lives in `phase-0.md` through `phase-11.md`;
[`summary.md`](summary.md) gives the one-page overview and status of all of
them.*

Ten phases. Each one: concepts first, then the build, then a checkable exit
condition. Total time-boxed at roughly **4–6 weeks part-time** (evenings +
weekends). Nothing here is theory for theory's sake — every concept block is
followed immediately by the place it lands in the code.

---

## Read this first — the four questions blocking you

### "What *is* an APM? Do you install it, use it, implement it, or build it?"

APM (Application Performance Monitoring) is a **product category and a capability**,
not a protocol or a single tool. A system has APM when you can answer, for
production traffic: what's the request rate, error rate and latency distribution
per endpoint; which services call which; where did *this specific* request spend its
time; which database query or external call was slow; and what error was thrown,
with the stack.

The verb depends on the vendor model:

- **Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM).** You
  *install* an agent that attaches to your runtime and ships to their backend. Mostly
  a deployment exercise. Turnkey, expensive, proprietary.
- **Grafana stack — what Babak asked for.** There is no product called "Grafana APM"
  that you install. You **assemble** the capability from four layers:

  | Layer | What you do | Component |
  |---|---|---|
  | Instrumentation | *Implement* — this is code you write | OpenTelemetry SDK inside your services |
  | Pipeline | *Deploy + configure* | OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution) |
  | Storage | *Deploy* | Tempo (traces), Loki (logs), Prometheus or Mimir (metrics) |
  | Query & UI | *Configure* | Grafana + provisioned datasources with cross-links |

  So: you **implement** instrumentation, you **deploy** the pipeline and stores, you
  **configure** the correlation. The "APM" is the emergent property of all four
  working together.

The distinction that actually matters: APM data is generated *inside* the
application, keyed to *individual requests*, and *correlated* across signals.
Infrastructure monitoring (CPU, disk, pod restarts) is generated *outside* the
application and aggregated. You need both; only the first is APM.

### "What's the difference between Logs, Metrics, Events and Traces?"

The clean way to hold this in your head is: **an event is the atom; the other three
are shapes you give it.**

| | What it is | Cost shape | Answers |
|---|---|---|---|
| **Event** | A structured record of one thing that happened, at one instant, with arbitrary attributes attached | Grows with traffic × attribute count | "What exactly happened here?" |
| **Log** | An event rendered for storage/search. A *structured* log is literally an event; an unstructured log is an event you threw the schema away from | Grows with traffic | "Why did it fail?" |
| **Metric** | A pre-aggregated numeric measurement over many events, sliced by a small set of labels | Grows with *label cardinality*, ~flat in traffic | "Is it broken? How much? Since when?" |
| **Trace** | A causally linked tree of events that each have a *duration* and a parent — a span tree spanning process boundaries | Grows with traffic × span count | "Where did the time go, across services?" |

Three consequences worth internalising now:

1. **Metrics are cheap in traffic but expensive in cardinality.** Doubling your
   request rate barely changes Prometheus's memory; adding `user_id` as a label
   can kill it. Logs and traces are the opposite: cheap in dimensions, expensive
   in volume. This asymmetry drives every design decision later.
2. **Traces are events with two extra fields:** a duration and a parent pointer.
   That's the whole trick. Everything else (service maps, critical paths, span
   metrics) is derived from those two fields.
3. In OpenTelemetry, "events" aren't a fourth top-level signal you export
   separately. They show up as **span events** (a timestamped annotation attached
   to a span — this is how `record_exception` works) and as **log records carrying
   an `event.name` attribute**; the Events API is built on the Logs data model.
   When Babak says M.E.L.T., he most likely means "logs plus the discrete
   business/domain events you care about" — check with him, but treat it as
   *structured logs and span events*, not a separate pipeline.

### "Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?"

**OpenTelemetry first, without much doubt.**

Loki, Tempo, Prometheus and Mimir are *destinations*. They're databases with query
languages. OpenTelemetry is the **producer-side data model and wire protocol** — it's
the only part of this that touches your Python code, and it's the part that
determines what the destinations can possibly show you. If you learn Loki first you
learn Loki's label model and its promtail-isms, most of which don't transfer. If you
learn OTel first, every backend becomes "point OTLP at a different URL."

One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
early, in parallel with OTel metrics — because it *constrains what you're allowed to
emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
(Phases 5–6), not up front.

Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
part of the Grafana stack and you do not need them for this task. Phase 10 covers
them as a judgment question, which is exactly how Babak framed them.

### Verdict on the OneUptime article you found

**Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
teaching resource. Read it in **Phase 5/6**, not now; it assumes you already
understand spans, span metrics and PromQL.

What's genuinely good in it, and rare to find in one place:

- The `spanmetrics` and `servicegraph` **connectors** in the Collector — this is how
  you get RED metrics and a service map "for free" from traces you already emit.
  This is the single highest-leverage config in the whole article.
- The Tempo datasource provisioning YAML with `tracesToLogsV2`, `tracesToMetrics`,
  `serviceMap` and `nodeGraph` — the click-through correlation that makes Grafana
  feel like an APM.
- `exemplarTraceIdDestinations` on the Prometheus datasource — jump from a latency
  spike on a graph straight into the trace that caused it.
- The tail-sampling policy block (keep all errors, keep all slow traces, sample the
  rest).

Two caveats:

- Its Collector config exports logs via the **`loki` exporter**. Loki now accepts
  **OTLP natively** on its own endpoint, and that is the current recommended path —
  fewer moving parts, better attribute mapping. Use the native OTLP route.
- It's a vendor blog (OneUptime sells a competing product). The Grafana content is
  accurate, but treat the official OpenTelemetry and Grafana docs as the source of
  truth for anything spec-level.
- **Its `spanmetrics` / `servicegraph` Collector connectors do not apply to your
  stack.** See the next section — your platform generates those metrics in Tempo
  instead. The article's PromQL is still useful, but the metric names differ.

---

## The platform Babak gave you (`observability-ansible`)

This changes *where* your telemetry goes and *how the pipeline is configured*. It
does not change the learning arc, the app, or the test suite. Read this before
Phase 2; it rewrites parts of Phases 2, 4, 5, 6, 8 and 10.

### What the repo is

An Ansible project that deploys **two independent observability backends** on one
Ubuntu/Debian host via Docker Compose:

1. **ELK + Grafana** — Filebeat → Logstash → Elasticsearch → Kibana/Grafana
2. **Grafana stack** — Prometheus + Alertmanager + Loki + Tempo + Grafana, with
   **Grafana Alloy** as the collector and node-exporter for host metrics

It contains **no application code and no dashboards**. It is the *platform half* of
Babak's task, handed to you pre-built. Your half is everything that produces
telemetry and everything that reads it.

Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
`playbooks/elk-grafana.yml`, idempotent, versions pinned in `group_vars/all.yml`
(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
Elastic 9.1.3). Everything lands under `/opt/observability/`.

> **How it actually runs here (since 2026-10-12):** the Grafana stack is ported
> to the k3s cluster `falken` as **`k8s/`** (same versions, configs and pipeline)
> and deployed with `kubectl apply -k k8s/`; the Ansible project stays in
> `ansible/` as the reference. Everything below describes the Ansible design and
> still holds for the port. Where it names a template, the k8s equivalent is:
> `config.alloy.j2` → `k8s/alloy/config.alloy` (plus the Docker-log half in
> `observability/alloy-agent/config.alloy` on czy), `prometheus.yml.j2` /
> `prometheus_extra_scrape_jobs` → `k8s/prometheus/prometheus.yml`,
> `tempo.yml.j2` → `k8s/tempo/tempo.yml`, `datasources.yml.j2` →
> `k8s/grafana/datasources.yml`, `alertmanager.yml.j2` →
> `k8s/alertmanager/alertmanager.yml`, "re-run the playbook" →
> `kubectl apply -k k8s/`. Deviations are listed in `k8s/README.md`. The ELK
> stack is not ported.

### The architecture

Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
**Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline →
Explore & Correlate**, with three input classes feeding it.

```
App metrics       ──/metrics, Prometheus exposition──┐
Container logs    ──Docker socket, tail──▶ Alloy ──push API──▶ Loki
Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo
                     ("spans + resource attrs")                    │
Node Exporter :9100 ──────scrape──────┐                            │
                                      ▼                            │
                                 Prometheus ◀──remote_write────────┘
                                  (TSDB, PromQL, rules)   span metrics + exemplars
                                      │
                             firing alerts ▼
                                 Alertmanager ──route──▶ Receivers
                                  (group, dedupe)      (Slack, Email, PagerDuty,
                                      │                 webhook, SMTP)
       Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four
              Correlation key: trace_id / exemplars
```

Note what the diagram says about the OTLP box: **"spans + resource attrs"**. Traces
only. And app metrics are labelled **"/metrics endpoint, Prometheus exposition"** —
not OTLP. The diagram and the code agree on this; it is a design decision, not an
oversight. Plan your instrumentation around it.

### One place the diagram and the code disagree

The diagram draws app metrics as `App metrics → scrape → Alloy → /metrics →
Prometheus`. The shipped config does not do that:

- `config.alloy.j2` contains **no `prometheus.scrape` component at all** — Alloy
  collects logs and receives traces, nothing else.
- `prometheus.yml.j2` scrapes `prometheus`, `node-exporter`, `loki`, `tempo`,
  `alloy:12345`, `grafana`, then loops over `prometheus_extra_scrape_jobs`. So your
  app targets are scraped by **Prometheus directly**, bypassing Alloy entirely. The
  green `/metrics` arrow from Alloy to Prometheus in the diagram is Prometheus
  scraping Alloy's **own** self-monitoring endpoint, not your app's metrics
  passing through it.

Neither is wrong; the diagram shows the intended architecture and the code
implements a simpler one. Closing that gap — adding `prometheus.scrape` +
`prometheus.remote_write` to Alloy so metrics really do flow through the collector —
is a genuinely good Phase 5 exercise, and **noticing the discrepancy at all is worth
mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
exactly the skill this task is meant to build.

### Four things in it that will cost you a day if you don't know them

**1. Alloy's OTLP receiver only forwards traces.** The config is literally:

```
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}
```

There is **no `metrics` output and no `logs` output**, and Alloy only registers
an OTLP/HTTP route for a signal that has one. So OTLP metrics and OTLP logs your
app sends get a **404** from `/v1/metrics` and `/v1/logs` (verified live
2026-10-12). The only evidence is an export error in your app's own logs, which
is easy to miss. Nothing shows up in Alloy, Prometheus or Loki; data just never
arrives in Grafana. This is the single most likely thing to eat an evening.

The diagram confirms this is deliberate: the OTLP input is labelled "spans +
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
input. So you have three options, and you should understand all three:

- *Follow the platform's grain (simplest):* expose a Prometheus `/metrics` endpoint
  per service and register them in `prometheus_extra_scrape_jobs` in
  `group_vars/all.yml`. Prometheus scrapes your services directly. Logs continue to
  arrive via stdout scraping.
- *Implement what the diagram draws:* add `prometheus.scrape` +
  `prometheus.remote_write` components to Alloy so app metrics flow through the
  collector, as the architecture intends. Buys you central relabeling and one place
  to change collection policy.
- *Go all-OTLP:* add `metrics` and `logs` outputs to the receiver, plus
  `otelcol.exporter.prometheus` → `prometheus.remote_write` and
  `otelcol.exporter.loki` → `loki.write`. One protocol for all three signals — the
  architecture you'd argue for on a greenfield platform, and the furthest from what
  ships.

Do the first to get moving, then do at least one of the others as a deliberate
exercise and write up the trade-off. That comparison is worth more than any single
answer.

**2. Span metrics come from Tempo, not from the collector.** `tempo.yml.j2` enables:

```
metrics_generator:
  storage:
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]
```

So RED metrics, the service graph, and exemplars already exist — generated inside
Tempo and remote-written into Prometheus (which runs with
`--web.enable-remote-write-receiver`). **Do not add `spanmetrics`/`servicegraph`
connectors to Alloy** — you'd be generating the same series twice under different
names. The metric names differ from the OneUptime article's; go find the actual
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.)

*Verified live on 2026-10-12: as shipped, none of this worked.* `local-blocks`
needs `metrics_generator.traces_storage`. Without it Tempo 2.8 logs "local blocks
processor requires traces wal" and starts **no** processors, so there are no span
metrics and no service graph. Separately, Prometheus throws away remote-written
exemplars unless it runs with `--enable-feature=exemplar-storage`. Both are fixed
in `k8s/`. It's worth telling Babak, and it's a good example of "config present
≠ data arriving".

**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
datasource is provisioned with:

```
derivedFields:
  - datasourceUid: tempo
    matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
    name: TraceID
```

Your log lines must contain **lowercase** `trace_id` (or `traceid` / `trace-id`).
This matters more than it looks: OpenTelemetry's `opentelemetry-instrumentation-logging`
auto-injection emits **`otelTraceID`**, which does **not** match that pattern.
So on this platform, the structlog processor emitting a lowercase `trace_id` field
isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus
datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram states
this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
in the Explore & Correlate column hangs off that one field name.

*Verified live:* Tempo's metrics-generator labels the exemplars it writes
**`traceID`**, not `trace_id`. So the provisioned exemplar destination does not
match the span-metric exemplars the platform itself produces, and clicking one
won't open the trace. This is left unresolved on purpose (Phase 5 material).
Work out the options and pick one.

**4. There is no dashboard provisioning at all.** The Grafana provisioning mount
contains only `datasources/`. Every dashboard is yours to add — which means Babak's
"create dashboards as code, destroy and recreate in seconds" requirement is entirely
your deliverable, and the natural home for it is *next to the platform's own
provisioning*: originally the Ansible role, now `k8s/grafana/`.

### What this changes, phase by phase

| Phase | Impact |
|---|---|
| 0 / 0a | **None.** App and test suite are unaffected. |
| 1 | **New.** Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes. |
| 2 | Replaced. Deploy the real stack (ported to k3s as `k8s/`) instead of running a single container. |
| 3 | Endpoint changes to Alloy; tracing work itself is identical. |
| 4 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
| 5 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
| 6 | **Upgraded.** Dashboards-as-code ships with the platform (`k8s/grafana/`, rebuilt by `kubectl apply -k k8s/`) — a better answer than a Makefile target. |
| 7 | None. |
| 8 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
| 9 | None, plus TraceQL metrics from `local-blocks`. |
| 10 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
| 11 | Demo now includes `kubectl apply -k k8s/` rebuilding the platform. |

### Where it runs

The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
ELK. Ledgerline runs on `czy` (1 vCPU / 1 GB, Phase 1), which has no room for
either. The Grafana stack therefore runs on the single-node k3s cluster `falken`
(4 CPU / 8 GB, shared with other workloads) in namespace
`ledgerline-observability`, with resource requests and limits set. ELK alongside
it would hurt; decide where it goes when Phase 10 comes.

In the Ansible design, Alloy's `loki.source.docker` reads the *platform host's*
Docker socket. That only works when the apps run on that host. Here they don't,
so that half of the config runs as a small Alloy agent on `czy`
(`observability/alloy-agent/config.alloy`). It picks up **every** container's
stdout there, including Traefik, Postgres and Redis, and pushes it to
`https://loki.vione.cloud`.

### One thing to confirm with Babak

Is this repo a **platform you must consume as-is**, or a **starting point you may
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
assumes you own it and will edit the Alloy config, add a dashboards provider, and
add scrape jobs. If Babak intends it as a fixed platform, several
recommendations below change — ask before Phase 4. Porting it to k3s (and the two
config fixes) is already a modification worth mentioning to him.

---

## Your resource spine

Pick **one paid course as a backbone** and use the free material as targeted
supplements. Both Udemy options below ship captions, which matters for your
subtitle-to-notes workflow.

**Backbone — choose one:**

| Resource | Format | Length | Cost | Best for |
|---|---|---|---|---|
| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 4–6. Comes with browser-based Killercoda playgrounds. |
| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 3–5. |

If budget allows, buy both — they overlap by maybe 30% and the second half of each
is different. If only one: take the **OpenTelemetry** one, because instrumentation
is the part you have to *write*, and Grafana's own docs are excellent for the rest.

**Free, and worth your time:**

| Resource | Format | Length | Use in |
|---|---|---|---|
| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 2. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 3. The best Python-specific overview, with a companion repo. |
| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 3. Auto- vs manual-instrumentation side by side. |
| **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 2. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 2–5. The neutral source of truth. |
| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 6–7 |
| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 2–5. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 5, 9. This is where your RED metrics and service graph actually come from. |
| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 10, now hands-on |
| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 8 |
| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 8. Self-run the exercises with Ledgerline as the subject service. |
| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 8, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 9 |
| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 5. The clearest explanation anywhere of what cardinality actually costs. |
| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 3 |
| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 10 |

**Where to run it:** Docker Compose on your laptop for Phases 0–9. Don't put this on
your k8s cluster until everything works locally — debugging instrumentation and
debugging Kubernetes at the same time is how these projects die. Porting it to the
cluster is a good Phase 11+ follow-up, and it's the natural bridge to the DevOps half
Babak has planned for you.