LedgerLine/docs/roadmap/phase-2.md

# Phase 2 — The model, and one signal end to end

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*

**Status:** Not started

**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
described above. Waiting to begin.

**Time-box:** one evening.

**Why:** you need the vocabulary, but more importantly you need to see one trace
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
than abstract.

### Concepts

- **Monitoring vs observability.** Monitoring answers questions you knew to ask
  (known unknowns): dashboards and alerts built from predefined aggregates.
  Observability is a property of the system: can you ask a *new* question about
  production, that you never anticipated, without shipping new code? Charity Majors'
  framing — monitoring is known-unknowns, observability is unknown-unknowns. The
  practical implication for you: emit **wide, high-dimensional events** (rich span
  attributes), not just pre-aggregated counters.
- **The OTel architecture split: API vs SDK vs Collector.**
  - The **API** is what your code and third-party instrumentation libraries call.
    If no SDK is configured, API calls are no-ops — that's deliberate, it means a
    library can be instrumented without forcing a dependency on any implementation.
  - The **SDK** is the engine: sampling, batching, context propagation, export.
    Application owners configure the SDK; library authors only touch the API.
  - The **Collector** is a standalone process that receives, processes and exports
    telemetry. Optional, but you should use one.
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
  `service.version`, `deployment.environment`) and is attached to every signal from
  that process. Attributes describe the individual span/metric/log.
- **Semantic conventions**: standardised attribute names (`http.request.method`,
  `db.system`, `server.address`). Use them for anything standard; namespace your own
  (`ledgerline.account.id`). This is what makes cross-service, cross-language
  correlation work — and it's what Grafana's prebuilt dashboards key off.
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
  aware of, not learning yet.)
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
  app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
  seven services. The Collector also does the things you don't want in your app:
  retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
  destinations.

### Resources

- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
  M.E.L.T. and context-propagation sections; skip the Node code.
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
  repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
  it — just see what "done" looks like.

### Build

1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
   `ansible-galaxy collection install -r requirements.yml`, change
   `grafana_admin_password` (vault it), then
   `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
   README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
   Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
   Then open the Alloy UI and **read the component graph** — it shows you the exact
   pipeline, including the fact that the OTLP receiver has only a traces output.
   Seeing that in the UI before you get bitten by it is worth ten minutes.
   *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
   a single container with everything pre-wired. Useful for a first trace, but
   don't build on it — its pipeline differs from your real platform in exactly the
   ways that matter.)*
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
   auto-instrumentation:
   ```
   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
   ```
   and change its command to
   `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
   with `OTEL_SERVICE_NAME=sanctions`,
   `OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
   `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
   Tempo directly.
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.

**Done when:** you can see a single-service trace in Tempo, and you can explain out
loud what the Resource attributes on it mean and where they came from.