LedgerLine/docs/roadmap/phase-2.md

Phase 2 — The model, and one signal end to end

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-1.md · Next: phase-3.md

Status: Not started

Message: Blocked on nothing — Phases 0 and 1 are done and the platform is described above. Waiting to begin.

Time-box: one evening.

Why: you need the vocabulary, but more importantly you need to see one trace appear in Grafana within an hour, so the rest of the roadmap feels concrete rather than abstract.

Concepts

• Monitoring vs observability. Monitoring answers questions you knew to ask (known unknowns): dashboards and alerts built from predefined aggregates. Observability is a property of the system: can you ask a new question about production, that you never anticipated, without shipping new code? Charity Majors' framing — monitoring is known-unknowns, observability is unknown-unknowns. The practical implication for you: emit wide, high-dimensional events (rich span attributes), not just pre-aggregated counters.
• The OTel architecture split: API vs SDK vs Collector.
  • The API is what your code and third-party instrumentation libraries call. If no SDK is configured, API calls are no-ops — that's deliberate, it means a library can be instrumented without forcing a dependency on any implementation.
  • The SDK is the engine: sampling, batching, context propagation, export. Application owners configure the SDK; library authors only touch the API.
  • The Collector is a standalone process that receives, processes and exports telemetry. Optional, but you should use one.
• OTLP is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
• Resource vs attributes: a Resource describes the producer (service.name, service.version, deployment.environment) and is attached to every signal from that process. Attributes describe the individual span/metric/log.
• Semantic conventions: standardised attribute names (http.request.method, db.system, server.address). Use them for anything standard; namespace your own (ledgerline.account.id). This is what makes cross-service, cross-language correlation work — and it's what Grafana's prebuilt dashboards key off.
• Signals: traces, metrics, logs. (Profiling is a fourth signal, in alpha — aware of, not learning yet.)
• Why a Collector rather than exporting straight to Tempo/Loki/Prometheus: your app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of seven services. The Collector also does the things you don't want in your app: retry, PII redaction, tail sampling, memory limiting, fan-out to multiple destinations.

Resources

• freeCodeCamp OpenTelemetry Course (~1 h, free) — watch at 1.5×, take notes on the M.E.L.T. and context-propagation sections; skip the Node code.
• OpenTelemetry docs → Concepts → Components and Signals. ~30 min.
• Optional: run the OpenTelemetry Demo (docker compose up in the astronomy-shop repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand it — just see what "done" looks like.

Build

1. Deploy the platform. Provision a VM, point inventory/hosts.yml at it, ansible-galaxy collection install -r requirements.yml, change grafana_admin_password (vault it), then ansible-playbook playbooks/grafana-stack.yml. Verify every endpoint from the README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093, Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318. Then open the Alloy UI and read the component graph — it shows you the exact pipeline, including the fact that the OTLP receiver has only a traces output. Seeing that in the UI before you get bitten by it is worth ten minutes. (Optional laptop fallback while the VM is being set up: grafana/otel-lgtm, a single container with everything pre-wired. Useful for a first trace, but don't build on it — its pipeline differs from your real platform in exactly the ways that matter.)
2. Pick the sanctions service (the smallest) and add nothing but zero-code auto-instrumentation:

   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
   

   and change its command to opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003 with OTEL_SERVICE_NAME=sanctions, OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf — pointing at Alloy, not at Tempo directly.
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.

Done when: you can see a single-service trace in Tempo, and you can explain out loud what the Resource attributes on it mean and where they came from.