LedgerLine/docs/roadmap/intro.md

Observability & APM — Delivery Roadmap for Babak's Task

Part of docs/roadmap/ — platform background and reading guidance shared by every phase. Per-phase detail lives in phase-0.md through phase-11.md; summary.md gives the one-page overview and status of all of them.

Ten phases. Each one: concepts first, then the build, then a checkable exit condition. Total time-boxed at roughly 4–6 weeks part-time (evenings + weekends). Nothing here is theory for theory's sake — every concept block is followed immediately by the place it lands in the code.

---

Read this first — the four questions blocking you

"What is an APM? Do you install it, use it, implement it, or build it?"

APM (Application Performance Monitoring) is a product category and a capability, not a protocol or a single tool. A system has APM when you can answer, for production traffic: what's the request rate, error rate and latency distribution per endpoint; which services call which; where did this specific request spend its time; which database query or external call was slow; and what error was thrown, with the stack.

The verb depends on the vendor model:

• Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM). You install an agent that attaches to your runtime and ships to their backend. Mostly a deployment exercise. Turnkey, expensive, proprietary.

• Grafana stack — what Babak asked for. There is no product called "Grafana APM" that you install. You assemble the capability from four layers:

  Layer	What you do	Component
  Instrumentation	Implement — this is code you write	OpenTelemetry SDK inside your services
  Pipeline	Deploy + configure	OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution)
  Storage	Deploy	Tempo (traces), Loki (logs), Prometheus or Mimir (metrics)
  Query & UI	Configure	Grafana + provisioned datasources with cross-links

  So: you implement instrumentation, you deploy the pipeline and stores, you configure the correlation. The "APM" is the emergent property of all four working together.

The distinction that actually matters: APM data is generated inside the application, keyed to individual requests, and correlated across signals. Infrastructure monitoring (CPU, disk, pod restarts) is generated outside the application and aggregated. You need both; only the first is APM.

"What's the difference between Logs, Metrics, Events and Traces?"

The clean way to hold this in your head is: an event is the atom; the other three are shapes you give it.

	What it is	Cost shape	Answers
Event	A structured record of one thing that happened, at one instant, with arbitrary attributes attached	Grows with traffic × attribute count	"What exactly happened here?"
Log	An event rendered for storage/search. A structured log is literally an event; an unstructured log is an event you threw the schema away from	Grows with traffic	"Why did it fail?"
Metric	A pre-aggregated numeric measurement over many events, sliced by a small set of labels	Grows with label cardinality, ~flat in traffic	"Is it broken? How much? Since when?"
Trace	A causally linked tree of events that each have a duration and a parent — a span tree spanning process boundaries	Grows with traffic × span count	"Where did the time go, across services?"

Three consequences worth internalising now:

1. Metrics are cheap in traffic but expensive in cardinality. Doubling your request rate barely changes Prometheus's memory; adding user_id as a label can kill it. Logs and traces are the opposite: cheap in dimensions, expensive in volume. This asymmetry drives every design decision later.
2. Traces are events with two extra fields: a duration and a parent pointer. That's the whole trick. Everything else (service maps, critical paths, span metrics) is derived from those two fields.
3. In OpenTelemetry, "events" aren't a fourth top-level signal you export separately. They show up as span events (a timestamped annotation attached to a span — this is how record_exception works) and as log records carrying an event.name attribute; the Events API is built on the Logs data model. When Babak says M.E.L.T., he most likely means "logs plus the discrete business/domain events you care about" — check with him, but treat it as structured logs and span events, not a separate pipeline.

"Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?"

OpenTelemetry first, without much doubt.

Loki, Tempo, Prometheus and Mimir are destinations. They're databases with query languages. OpenTelemetry is the producer-side data model and wire protocol — it's the only part of this that touches your Python code, and it's the part that determines what the destinations can possibly show you. If you learn Loki first you learn Loki's label model and its promtail-isms, most of which don't transfer. If you learn OTel first, every backend becomes "point OTLP at a different URL."

One exception: learn Prometheus's data model (labels, series, cardinality) fairly early, in parallel with OTel metrics — because it constrains what you're allowed to emit. PromQL, LogQL and TraceQL are query languages; learn them when you need them (Phases 5–6), not up front.

Filebeat and Logstash are a different world entirely (Elastic's). They are not part of the Grafana stack and you do not need them for this task. Phase 10 covers them as a judgment question, which is exactly how Babak framed them.

Verdict on the OneUptime article you found

Yes, it's valuable — but not yet. It's a configuration cheat-sheet, not a teaching resource. Read it in Phase 5/6, not now; it assumes you already understand spans, span metrics and PromQL.

What's genuinely good in it, and rare to find in one place:

• The spanmetrics and servicegraph connectors in the Collector — this is how you get RED metrics and a service map "for free" from traces you already emit. This is the single highest-leverage config in the whole article.
• The Tempo datasource provisioning YAML with tracesToLogsV2, tracesToMetrics, serviceMap and nodeGraph — the click-through correlation that makes Grafana feel like an APM.
• exemplarTraceIdDestinations on the Prometheus datasource — jump from a latency spike on a graph straight into the trace that caused it.
• The tail-sampling policy block (keep all errors, keep all slow traces, sample the rest).

Two caveats:

• Its Collector config exports logs via the loki exporter. Loki now accepts OTLP natively on its own endpoint, and that is the current recommended path — fewer moving parts, better attribute mapping. Use the native OTLP route.
• It's a vendor blog (OneUptime sells a competing product). The Grafana content is accurate, but treat the official OpenTelemetry and Grafana docs as the source of truth for anything spec-level.
• Its spanmetrics / servicegraph Collector connectors do not apply to your stack. See the next section — your platform generates those metrics in Tempo instead. The article's PromQL is still useful, but the metric names differ.

---

The platform Babak gave you (observability-ansible)

This changes where your telemetry goes and how the pipeline is configured. It does not change the learning arc, the app, or the test suite. Read this before Phase 2; it rewrites parts of Phases 2, 4, 5, 6, 8 and 10.

What the repo is

An Ansible project that deploys two independent observability backends on one Ubuntu/Debian host via Docker Compose:

1. ELK + Grafana — Filebeat → Logstash → Elasticsearch → Kibana/Grafana
2. Grafana stack — Prometheus + Alertmanager + Loki + Tempo + Grafana, with Grafana Alloy as the collector and node-exporter for host metrics

It contains no application code and no dashboards. It is the platform half of Babak's task, handed to you pre-built. Your half is everything that produces telemetry and everything that reads it.

Deployment: ansible-playbook playbooks/grafana-stack.yml and playbooks/elk-grafana.yml, idempotent, versions pinned in group_vars/all.yml (Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2, Elastic 9.1.3). Everything lands under /opt/observability/.

The architecture

Per diagrams/02-grafana-lgtm-operational.svg, the platform is five columns: Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline → Explore & Correlate, with three input classes feeding it.

App metrics       ──/metrics, Prometheus exposition──┐
Container logs    ──Docker socket, tail──▶ Alloy ──push API──▶ Loki
Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo
                     ("spans + resource attrs")                    │
Node Exporter :9100 ──────scrape──────┐                            │
                                      ▼                            │
                                 Prometheus ◀──remote_write────────┘
                                  (TSDB, PromQL, rules)   span metrics + exemplars
                                      │
                             firing alerts ▼
                                 Alertmanager ──route──▶ Receivers
                                  (group, dedupe)      (Slack, Email, PagerDuty,
                                      │                 webhook, SMTP)
       Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four
              Correlation key: trace_id / exemplars

Note what the diagram says about the OTLP box: "spans + resource attrs". Traces only. And app metrics are labelled "/metrics endpoint, Prometheus exposition" — not OTLP. The diagram and the code agree on this; it is a design decision, not an oversight. Plan your instrumentation around it.

One place the diagram and the code disagree

The diagram draws app metrics as App metrics → scrape → Alloy → /metrics → Prometheus. The shipped config does not do that:

• config.alloy.j2 contains no prometheus.scrape component at all — Alloy collects logs and receives traces, nothing else.
• prometheus.yml.j2 scrapes prometheus, node-exporter, loki, tempo, alloy:12345, grafana, then loops over prometheus_extra_scrape_jobs. So your app targets are scraped by Prometheus directly, bypassing Alloy entirely. The green /metrics arrow from Alloy to Prometheus in the diagram is Prometheus scraping Alloy's own self-monitoring endpoint, not your app's metrics passing through it.

Neither is wrong; the diagram shows the intended architecture and the code implements a simpler one. Closing that gap — adding prometheus.scrape + prometheus.remote_write to Alloy so metrics really do flow through the collector — is a genuinely good Phase 5 exercise, and noticing the discrepancy at all is worth mentioning to Babak. Spotting that a diagram and its implementation have drifted is exactly the skill this task is meant to build.

Four things in it that will cost you a day if you don't know them

1. Alloy's OTLP receiver only forwards traces. The config is literally:

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

There is no metrics output and no logs output. OTLP metrics and OTLP logs your app sends are accepted and then silently discarded — no error, no warning, just nothing arriving in Grafana. This is the single most likely thing to eat an evening.

The diagram confirms this is deliberate: the OTLP input is labelled "spans + resource attrs", and metrics are drawn as a separate /metrics Prometheus-exposition input. So you have three options, and you should understand all three:

• Follow the platform's grain (simplest): expose a Prometheus /metrics endpoint per service and register them in prometheus_extra_scrape_jobs in group_vars/all.yml. Prometheus scrapes your services directly. Logs continue to arrive via stdout scraping.
• Implement what the diagram draws: add prometheus.scrape + prometheus.remote_write components to Alloy so app metrics flow through the collector, as the architecture intends. Buys you central relabeling and one place to change collection policy.
• Go all-OTLP: add metrics and logs outputs to the receiver, plus otelcol.exporter.prometheus → prometheus.remote_write and otelcol.exporter.loki → loki.write. One protocol for all three signals — the architecture you'd argue for on a greenfield platform, and the furthest from what ships.

Do the first to get moving, then do at least one of the others as a deliberate exercise and write up the trade-off. That comparison is worth more than any single answer.

2. Span metrics come from Tempo, not from the collector. tempo.yml.j2 enables:

metrics_generator:
  storage:
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]

So RED metrics, the service graph, and exemplars already exist — generated inside Tempo and remote-written into Prometheus (which runs with --web.enable-remote-write-receiver). Do not add spanmetrics/servicegraph connectors to Alloy — you'd be generating the same series twice under different names. The metric names differ from the OneUptime article's; go find the actual names in Prometheus before writing any PromQL. (local-blocks also gives you TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.)

3. Log→trace correlation depends on an exact, case-sensitive regex. The Loki datasource is provisioned with:

derivedFields:
  - datasourceUid: tempo
    matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
    name: TraceID

Your log lines must contain lowercase trace_id (or traceid / trace-id). This matters more than it looks: OpenTelemetry's opentelemetry-instrumentation-logging auto-injection emits otelTraceID, which does not match that pattern. So on this platform, the structlog processor emitting a lowercase trace_id field isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus datasource declares exemplarTraceIdDestinations: name: trace_id. The diagram states this outright in its legend: "Correlation key: trace_id / exemplars." Everything in the Explore & Correlate column hangs off that one field name.

4. There is no dashboard provisioning at all. The Grafana provisioning mount contains only datasources/. Every dashboard is yours to add — which means Babak's "create dashboards as code, destroy and recreate in seconds" requirement is entirely your deliverable, and the natural home for it is inside this Ansible role.

What this changes, phase by phase

Phase	Impact
0 / 0a	None. App and test suite are unaffected.
1	New. Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes.
2	Replaced. Deploy the real stack with Ansible instead of running a single container.
3	Endpoint changes to Alloy; tracing work itself is identical.
4	Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase trace_id is mandatory.
5	Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension.
6	Upgraded. Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target.
7	None.
8	Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting.
9	None, plus TraceQL metrics from local-blocks.
10	Massively upgraded. You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment.
11	Demo now includes ansible-playbook rebuilding the platform.

Where to run it

The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside everything else will hurt. Use a separate VM (a cheap cloud instance or a local Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing at it. That also makes the deployment real rather than a localhost shortcut, which is the point of it being Ansible.

If you do colocate, note that Alloy's loki.source.docker reads the host's Docker socket and will scrape every container's stdout, including Ledgerline's, automatically — convenient, and worth knowing before you wonder where those logs came from.

One thing to confirm with Babak

Is this repo a platform you must consume as-is, or a starting point you may modify? It's in your own Gitea and nothing marks it read-only, so this roadmap assumes you own it and will edit config.alloy.j2, add a dashboards provider, and extend group_vars. If Babak intends it as a fixed platform, several recommendations below change — ask before Phase 4.

---

Your resource spine

Pick one paid course as a backbone and use the free material as targeted supplements. Both Udemy options below ship captions, which matters for your subtitle-to-notes workflow.

Backbone — choose one:

Resource	Format	Length	Cost	Best for
Observability with Grafana, Prometheus, Loki, Alloy and Tempo (Aref Karimi, Udemy)	Video	~10 h, 132 lectures	Paid (often ~€15 on sale)	The Grafana-stack backbone. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 4–6. Comes with browser-based Killercoda playgrounds.
OpenTelemetry for Observability: The Complete Course (Udemy)	Video	~7.5 h, 113 lectures	Paid	The OTel backbone. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 3–5.

If budget allows, buy both — they overlap by maybe 30% and the second half of each is different. If only one: take the OpenTelemetry one, because instrumentation is the part you have to write, and Grafana's own docs are excellent for the rest.

Free, and worth your time:

Resource	Format	Length	Use in
freeCodeCamp — OpenTelemetry Course (Ania Kubów)	YouTube	~1 h	Phase 2. Concepts, M.E.L.T., context propagation. Note: the code is Node.js, not Python — watch it for the model, not the syntax.
Jessica Garson — Introduction to OpenTelemetry with Python (PyTexas 2025)	YouTube talk	~30 min	Phase 3. The best Python-specific overview, with a companion repo.
Jessica Garson — Introduction to OpenTelemetry with Django (DjangoCon US 2024)	YouTube talk	~30 min	Phase 3. Auto- vs manual-instrumentation side by side.
OpenTelemetry Demo ("Astronomy Shop") — github.com/open-telemetry/opentelemetry-demo, and Grafana's fork grafana/lgtm-otel-demo	Running code	—	Every phase. This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup.
grafana/docker-otel-lgtm Docker image	Tool	—	Phase 2. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in one container with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation.
Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling	Docs	—	Phases 2–5. The neutral source of truth.
Grafana docs — Observability as code / Foundation SDK; k6 docs	Docs	—	Phases 6–7
Grafana Alloy docs — components reference, especially otelcol.receiver.otlp, otelcol.exporter.*, loki.source.docker, prometheus.remote_write	Docs	—	Phases 2–5. Required — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines.
Grafana Tempo docs — Metrics-generator and TraceQL	Docs	—	Phases 5, 9. This is where your RED metrics and service graph actually come from.
Elastic docs — Filebeat inputs/modules, Logstash grok filter	Docs	—	Phase 10, now hands-on
Google SRE Workbook — Implementing SLOs, Alerting on SLOs, Error Budget Policy	Free book chapters	~2 h total	Phase 8
Google SRE — The Art of SLOs workshop (slides + participant & facilitator handbooks, all public PDFs)	Slides/PDF + exercises	~2 h	Phase 8. Self-run the exercises with Ledgerline as the subject service.
Coursera — Measuring and Managing Reliability (Google Cloud)	Video course	~1 week	Phase 8, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free.
Brendan Gregg — the USE method; Tom Wilkie — the RED method	Articles	~20 min each	Phase 9
Cloudflare — How Cloudflare runs Prometheus at scale	Article	~20 min	Phase 5. The clearest explanation anywhere of what cardinality actually costs.
SigNoz — OpenTelemetry FastAPI Instrumentation; Uptrace — OpenTelemetry FastAPI	Articles	~20 min each	Phase 3
Better Stack — Filebeat vs Logstash; SigNoz — OpenTelemetry vs ELK	Articles	~30 min	Phase 10

Where to run it: Docker Compose on your laptop for Phases 0–9. Don't put this on your k8s cluster until everything works locally — debugging instrumentation and debugging Kubernetes at the same time is how these projects die. Porting it to the cluster is a good Phase 11+ follow-up, and it's the natural bridge to the DevOps half Babak has planned for you.