LedgerLine/docs/roadmap/phase-2.md

Phase 2 — The model, and one signal end to end

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-1.md · Next: phase-3.md

Status: Done

Message: Started and finished 2026-10-12. The platform is deployed and the Alloy component graph has been read. sanctions runs under zero-code auto-instrumentation (opentelemetry-distro==0.66b1 + opentelemetry-exporter-otlp==1.45.1, instrumentors installed by opentelemetry-bootstrap at build time). It exports traces only, over OTLP/HTTP to https://otlp.vione.cloud. First real trace: 9bfa05d03eabe3d3338a8672cf127465 (POST /internal/lookup), found with {resource.service.name="sanctions" && kind=server && span.http.route!="/healthz"}.

Found along the way: Alloy returns 404 for OTLP metrics and logs. They are not silently accepted, and the docs were corrected to say so.

Carried forward, all undecided:

• /healthz traces. The Docker healthcheck produces a trace every 5s and floods Tempo search. OTEL_PYTHON_FASTAPI_EXCLUDED_URLS would filter it.
• deployment.environment.name=prod is set in the base docker-compose.yml, so local runs would also be labelled prod.
• Guard test G10 (test_dependency_files_do_not_list_observability_libraries) is red by choice until the guard suite is retired in Phase 3.

Time-box: one evening.

Why: you need the vocabulary, but more importantly you need to see one trace appear in Grafana within an hour, so the rest of the roadmap feels concrete rather than abstract.

Concepts

• Monitoring vs observability. Monitoring answers questions you knew to ask (known unknowns): dashboards and alerts built from predefined aggregates. Observability is a property of the system: can you ask a new question about production, that you never anticipated, without shipping new code? Charity Majors' framing — monitoring is known-unknowns, observability is unknown-unknowns. The practical implication for you: emit wide, high-dimensional events (rich span attributes), not just pre-aggregated counters.
• The OTel architecture split: API vs SDK vs Collector.
  • The API is what your code and third-party instrumentation libraries call. If no SDK is configured, API calls are no-ops — that's deliberate, it means a library can be instrumented without forcing a dependency on any implementation.
  • The SDK is the engine: sampling, batching, context propagation, export. Application owners configure the SDK; library authors only touch the API.
  • The Collector is a standalone process that receives, processes and exports telemetry. Optional, but you should use one.
• OTLP is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
• Resource vs attributes: a Resource describes the producer (service.name, service.version, deployment.environment) and is attached to every signal from that process. Attributes describe the individual span/metric/log.
• Semantic conventions: standardised attribute names (http.request.method, db.system, server.address). Use them for anything standard; namespace your own (ledgerline.account.id). This is what makes cross-service, cross-language correlation work — and it's what Grafana's prebuilt dashboards key off.
• Signals: traces, metrics, logs. (Profiling is a fourth signal, in alpha — aware of, not learning yet.)
• Why a Collector rather than exporting straight to Tempo/Loki/Prometheus: your app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of seven services. The Collector also does the things you don't want in your app: retry, PII redaction, tail sampling, memory limiting, fan-out to multiple destinations.

Resources

• freeCodeCamp OpenTelemetry Course (~1 h, free) — watch at 1.5×, take notes on the M.E.L.T. and context-propagation sections; skip the Node code.
• OpenTelemetry docs → Concepts → Components and Signals. ~30 min.
• Optional: run the OpenTelemetry Demo (docker compose up in the astronomy-shop repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand it — just see what "done" looks like.

Build

1. Deploy the platform. Deployed 2026-10-12. The mentor's Ansible Grafana stack was ported to the k3s cluster falken as k8s/, keeping the same versions, configs and pipeline. Deploy with kubectl apply -k k8s/. k8s/README.md covers deploy/teardown, credentials and every deviation from the Ansible role.

   Why not the Ansible route on a VM:

   • czy (1 GB) can't host it. The platform README recommends 2 vCPU / 4 GB.
   • *.vione.cloud sits behind Cloudflare's proxy, which only forwards a few ports (80/443, 8080, 8443, …), so raw ports like :3000 never reach any origin. That's why ledgerline.vione.cloud:3000 showed nothing.

   Endpoints: each one is a subdomain behind Traefik. Everything except Grafana uses basic auth (k8s/.credentials).

   Grafana	https://grafana.vione.cloud
   Prometheus	https://prometheus.vione.cloud
   Alertmanager	https://alertmanager.vione.cloud
   Loki	https://loki.vione.cloud
   Tempo (query)	https://tempo.vione.cloud
   Alloy UI	https://alloy.vione.cloud
   OTLP/HTTP ingest	https://otlp.vione.cloud

   Container logs from czy reach Loki through an Alloy agent there (the alloy-agent service in docker-compose.traefik.yml).

   Verified live: all of the following worked, using a scratch trace sent from czy.

   • Every endpoint answers, and returns 401 without auth.
   • Prometheus scrapes all six platform targets.
   • Traces flow czy → Alloy → Tempo.
   • Tempo's metrics-generator emits traces_spanmetrics_* and traces_service_graph_* with exemplars.
   • czy container logs land in Loki.

   Two platform bugs, fixed in the port. As shipped, the Ansible role produced no span metrics or service graph (Tempo needs metrics_generator.traces_storage), and Prometheus discarded exemplars (missing --enable-feature=exemplar-storage).

   One issue left open on purpose: Tempo labels its exemplars traceID, but the datasource's exemplarTraceIdDestinations expects trace_id.

   Now open the Alloy UI and read the component graph. It shows you the exact pipeline, including the fact that the OTLP receiver has only a traces output. Seeing that in the UI before you get bitten by it is worth ten minutes.

2. Pick the sanctions service (the smallest) and add nothing but zero-code auto-instrumentation:

   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
   

   and change its command to opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003 with OTEL_SERVICE_NAME=sanctions, OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf — pointing at Alloy, not at Tempo directly — plus the basic-auth header OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 user:pass> (URL-encoded: the space is %20; see k8s/README.md). Keep the credential in the gitignored .env.observability on czy, not in compose.

3. Fire a request. Open Grafana → Explore → Tempo → find the trace.

Done when: you can see a single-service trace in Tempo, and you can explain out loud what the Resource attributes on it mean and where they came from.