LedgerLine/docs/roadmap/phase-2.md

# Phase 2 — The model, and one signal end to end

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*

**Status:** Done

**Message:** Started and finished 2026-10-12. The platform is deployed and the
Alloy component graph has been read. `sanctions` runs under zero-code
auto-instrumentation (`opentelemetry-distro==0.66b1` +
`opentelemetry-exporter-otlp==1.45.1`, instrumentors installed by
`opentelemetry-bootstrap` at build time). It exports traces only, over
OTLP/HTTP to `https://otlp.vione.cloud`. First real trace:
`9bfa05d03eabe3d3338a8672cf127465` (`POST /internal/lookup`), found with
`{resource.service.name="sanctions" && kind=server && span.http.route!="/healthz"}`.

Found along the way: Alloy returns **404** for OTLP metrics and logs. They are
not silently accepted, and the docs were corrected to say so.

Carried forward, all undecided:
- **`/healthz` traces.** The Docker healthcheck produces a trace every 5s and
  floods Tempo search. `OTEL_PYTHON_FASTAPI_EXCLUDED_URLS` would filter it.
- **`deployment.environment.name=prod`** is set in the base
  `docker-compose.yml`, so local runs would also be labelled `prod`.
- **Guard test G10** (`test_dependency_files_do_not_list_observability_libraries`)
  is red by choice until the guard suite is retired in Phase 3.

**Time-box:** one evening.

**Why:** you need the vocabulary, but more importantly you need to see one trace
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
than abstract.

### Concepts

- **Monitoring vs observability.** Monitoring answers questions you knew to ask
  (known unknowns): dashboards and alerts built from predefined aggregates.
  Observability is a property of the system: can you ask a *new* question about
  production, that you never anticipated, without shipping new code? Charity Majors'
  framing — monitoring is known-unknowns, observability is unknown-unknowns. The
  practical implication for you: emit **wide, high-dimensional events** (rich span
  attributes), not just pre-aggregated counters.
- **The OTel architecture split: API vs SDK vs Collector.**
  - The **API** is what your code and third-party instrumentation libraries call.
    If no SDK is configured, API calls are no-ops — that's deliberate, it means a
    library can be instrumented without forcing a dependency on any implementation.
  - The **SDK** is the engine: sampling, batching, context propagation, export.
    Application owners configure the SDK; library authors only touch the API.
  - The **Collector** is a standalone process that receives, processes and exports
    telemetry. Optional, but you should use one.
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
  `service.version`, `deployment.environment`) and is attached to every signal from
  that process. Attributes describe the individual span/metric/log.
- **Semantic conventions**: standardised attribute names (`http.request.method`,
  `db.system`, `server.address`). Use them for anything standard; namespace your own
  (`ledgerline.account.id`). This is what makes cross-service, cross-language
  correlation work — and it's what Grafana's prebuilt dashboards key off.
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
  aware of, not learning yet.)
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
  app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
  seven services. The Collector also does the things you don't want in your app:
  retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
  destinations.

### Resources

- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
  M.E.L.T. and context-propagation sections; skip the Node code.
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
  repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
  it — just see what "done" looks like.

### Build

1. **Deploy the platform.** *Deployed 2026-10-12.* The mentor's Ansible
   Grafana stack was ported to the k3s cluster `falken` as `k8s/`, keeping the
   same versions, configs and pipeline. Deploy with `kubectl apply -k k8s/`.
   `k8s/README.md` covers deploy/teardown, credentials and every deviation
   from the Ansible role.

   **Why not the Ansible route on a VM:**
   - `czy` (1 GB) can't host it. The platform README recommends 2 vCPU / 4 GB.
   - `*.vione.cloud` sits behind Cloudflare's proxy, which only forwards a few
     ports (80/443, 8080, 8443, …), so raw ports like `:3000` never reach any
     origin. That's why `ledgerline.vione.cloud:3000` showed nothing.

   **Endpoints:** each one is a subdomain behind Traefik. Everything except
   Grafana uses basic auth (`k8s/.credentials`).

   | | |
   |---|---|
   | Grafana | https://grafana.vione.cloud |
   | Prometheus | https://prometheus.vione.cloud |
   | Alertmanager | https://alertmanager.vione.cloud |
   | Loki | https://loki.vione.cloud |
   | Tempo (query) | https://tempo.vione.cloud |
   | Alloy UI | https://alloy.vione.cloud |
   | OTLP/HTTP ingest | https://otlp.vione.cloud |

   Container logs from `czy` reach Loki through an Alloy agent there (the
   `alloy-agent` service in `docker-compose.traefik.yml`).

   **Verified live:** all of the following worked, using a scratch trace sent
   from `czy`.
   - Every endpoint answers, and returns 401 without auth.
   - Prometheus scrapes all six platform targets.
   - Traces flow czy → Alloy → Tempo.
   - Tempo's metrics-generator emits `traces_spanmetrics_*` and
     `traces_service_graph_*` with exemplars.
   - czy container logs land in Loki.

   **Two platform bugs, fixed in the port.** As shipped, the Ansible role
   produced no span metrics or service graph (Tempo needs
   `metrics_generator.traces_storage`), and Prometheus discarded exemplars
   (missing `--enable-feature=exemplar-storage`).

   **One issue left open on purpose:** Tempo labels its exemplars `traceID`,
   but the datasource's `exemplarTraceIdDestinations` expects `trace_id`.

   Now open the Alloy UI and **read the component graph**. It shows you the
   exact pipeline, including the fact that the OTLP receiver has only a traces
   output. Seeing that in the UI before you get bitten by it is worth ten
   minutes.
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
   auto-instrumentation:
   ```
   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
   ```
   and change its command to
   `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
   with `OTEL_SERVICE_NAME=sanctions`,
   `OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud`,
   `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
   Tempo directly — plus the basic-auth header
   `OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 user:pass>`
   (URL-encoded: the space is `%20`; see `k8s/README.md`). Keep the
   credential in the gitignored `.env.observability` on czy, not in compose.
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.

**Done when:** you can see a single-service trace in Tempo, and you can explain out
loud what the Resource attributes on it mean and where they came from.