LedgerLine/docs/roadmap/phase-3.md

# Phase 3 — Distributed tracing

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*

**Status:** Done

**Message:** Started 2026-10-12, done 2026-10-13.
- `common/telemetry.py` bootstraps tracing in code for all seven services,
  replacing Phase 2's zero-code wrapper. It covers FastAPI, SQLAlchemy, httpx
  and Redis instrumentation.
- Trace context crosses the Redis Stream by hand: it's injected into the message
  inside a PRODUCER span and extracted by both consumer groups.
- Business spans were added, swallowed exceptions are recorded on their spans,
  and the response's `request_id` is now the trace ID. The guard suite is
  retired.
- One request produces one connected trace across all seven services, both
  consumers included (verified live during Phase 4: 51 spans, 7 services).
- The chaos-nightmare and "explain `blocking_event_loop`" criteria were attested
  by the user.

**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.

**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
(Resource, providers, exporters, Collector wiring) is shared across all three
signals — do it once here and logs and metrics become incremental. Second, tracing
is what Babak actually asked for ("trace a single request, see where it stuck"),
and once trace IDs exist, log correlation in Phase 4 works on the first try instead
of needing rework.

### Concepts

- **Trace / span / span context.** A trace is one request's journey; a span is one
  timed operation within it. Each span carries a trace ID, its own span ID, and its
  parent's span ID. That parent pointer is the entire data structure.
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
  outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
  these right is what makes the service graph render correctly.
- **Context propagation.**
  - *In-process*: OTel uses Python `contextvars`. This works across `await` — but
    **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
    carry the context explicitly.
  - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
    `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
  - The classic bug: any boundary that doesn't carry those headers **starts a brand
    new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
    `TraceContextTextMapPropagator().inject(...)` into the message payload on the
    producer and `.extract(...)` on the consumer, then starting the consumer span
    with that context as parent. This is the single most valuable exercise in the
    phase — most people never do it manually and never really understand it.
  - Related: `links` (for a consumer span that batches messages from many traces)
    and **Baggage** (arbitrary key/values propagated alongside trace context —
    useful, and a foot-gun, because it's sent on every hop and is visible to
    downstream systems; never put secrets in it).
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
  client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
  gets you the business semantics: `score_transaction`, `check_sanctions_list`,
  `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
  that show *which service* was slow but not *why*.
- **Span attributes vs metric labels.** Spans are the right home for
  high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
  with volume, not with distinct values. This is why "put it on the span, not the
  metric" is the standing rule.
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
  That's how the exception ends up attached to the trace and how the `spanmetrics`
  connector in Phase 5 will count your error rate.
- **Sampling — the concept now, the config later.**
  - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
    errors at the same rate as successes.
  - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
    traces, but must run in the Collector and buffer spans in memory.
  - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
    now; you'll configure tail sampling in the Collector in Phase 6.
  - Whatever you choose, the sampling decision must **propagate** — the sampled flag
    rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
    unilaterally drop half of a trace its caller decided to keep.

### Resources

- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
  min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
  practical FastAPI-specific setup, including batch/queue env vars.
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
  reading.
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
  side by side. Look specifically at how they mix auto-instrumentation with
  hand-rolled business spans, and how the Kafka producer/consumer propagate context.
  This is the "professional implementation" you asked to study.

### Build

1. **Shared bootstrap.** Create `common/telemetry.py` with a single
   `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
   `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
   service calls it once at startup. Keep it under 60 lines.
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
   `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
   you now get a connected trace across `gateway → ledger → fraud → sanctions`.
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
   `sanctions.lookup_watchlist`, `insights.aggregate_120d`,
   `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
   namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
   `ledgerline.tx.amount_minor`).
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
   inject trace context into the message fields. Consumers (`notifier`, `audit`):
   extract, and start a `CONSUMER` span with the extracted context as parent.
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
   more.
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
   the "before" state so you could measure against it, and from here it's
   actively wrong. Everything else in `tests/` must still pass: instrumenting
   the system is not allowed to change its behaviour, and if `make test` goes red
   after adding spans, you've broken something real. That is exactly the safety
   net you built it for.

### Done when

- One request produces **one connected trace** covering all seven services,
  including both async consumers.
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
  ate the time.
- You can explain why `blocking_event_loop` looks different in a trace than
  `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
  before spans start, plus every concurrent request in that process degrading
  together.)