LedgerLine/docs/roadmap/phase-2.md

Phase 2 — Distributed tracing

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-1.md · Next: phase-3.md

Status: Not started

Message: Depends on Phase 1's common/telemetry.py bootstrap existing first.

Time-box: 2–3 evenings. This is the biggest phase and the heart of the task.

Why traces before logs and metrics? Two reasons. First, the SDK bootstrap (Resource, providers, exporters, Collector wiring) is shared across all three signals — do it once here and logs and metrics become incremental. Second, tracing is what Babak actually asked for ("trace a single request, see where it stuck"), and once trace IDs exist, log correlation in Phase 3 works on the first try instead of needing rework.

Concepts

• Trace / span / span context. A trace is one request's journey; a span is one timed operation within it. Each span carries a trace ID, its own span ID, and its parent's span ID. That parent pointer is the entire data structure.
• Span kinds: SERVER (you handled an inbound request), CLIENT (you made an outbound call), PRODUCER / CONSUMER (async messaging), INTERNAL. Getting these right is what makes the service graph render correctly.
• Context propagation.
  • In-process: OTel uses Python contextvars. This works across await — but not across a bare threading.Thread or a ProcessPoolExecutor unless you carry the context explicitly.
  • Cross-process: the W3C Trace Context headers, traceparent and tracestate. (B3 is the legacy Zipkin format; you'll see it in older systems.)
  • The classic bug: any boundary that doesn't carry those headers starts a brand new trace. Your Redis Stream hop is exactly this. You'll fix it by hand with TraceContextTextMapPropagator().inject(...) into the message payload on the producer and .extract(...) on the consumer, then starting the consumer span with that context as parent. This is the single most valuable exercise in the phase — most people never do it manually and never really understand it.
  • Related: links (for a consumer span that batches messages from many traces) and Baggage (arbitrary key/values propagated alongside trace context — useful, and a foot-gun, because it's sent on every hop and is visible to downstream systems; never put secrets in it).
• Auto- vs manual instrumentation. Auto gets you the server span, the HTTP client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual gets you the business semantics: score_transaction, check_sanctions_list, settle_ledger_entry. You need both. Auto-instrumentation alone produces traces that show which service was slow but not why.
• Span attributes vs metric labels. Spans are the right home for high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale with volume, not with distinct values. This is why "put it on the span, not the metric" is the standing rule.
• Errors on spans: set span status to ERROR and call record_exception(e). That's how the exception ends up attached to the trace and how the spanmetrics connector in Phase 4 will count your error rate.
• Sampling — the concept now, the config later.
  • Head-based: decide at trace start. Cheap and predictable, but blind — it drops errors at the same rate as successes.
  • Tail-based: decide after the trace finishes. Can keep 100% of errors and slow traces, but must run in the Collector and buffer spans in memory.
  • For a local demo: sample 100% (ParentBased(AlwaysOn)). Understand the trade-off now; you'll configure tail sampling in the Collector in Phase 5.
  • Whatever you choose, the sampling decision must propagate — the sampled flag rides in traceparent. Use ParentBased samplers so a service doesn't unilaterally drop half of a trace its caller decided to keep.

Resources

• Jessica Garson — Introduction to OpenTelemetry with Python (PyTexas 2025, ~30 min, free) and …with Django (DjangoCon US 2024, ~30 min, free). Watch both.
• Your chosen Udemy backbone: the instrumentation + context-propagation sections.
• SigNoz — OpenTelemetry FastAPI Instrumentation (free article) — the most practical FastAPI-specific setup, including batch/queue env vars.
• OpenTelemetry docs → Python → Instrumentation and Propagation. Reference, not reading.
• OpenTelemetry Demo: open src/recommendation (Python) and the Go/TS services side by side. Look specifically at how they mix auto-instrumentation with hand-rolled business spans, and how the Kafka producer/consumer propagate context. This is the "professional implementation" you asked to study.

Build

1. Shared bootstrap. Create common/telemetry.py with a single setup_telemetry(service_name) that builds the Resource, TracerProvider, BatchSpanProcessor and OTLP exporter, and installs the propagator. Every service calls it once at startup. Keep it under 60 lines.
2. Auto-instrument all seven services: FastAPIInstrumentor, HTTPXClientInstrumentor, SQLAlchemyInstrumentor, RedisInstrumentor. Verify you now get a connected trace across gateway → ledger → fraud → sanctions.
3. Manual spans for business logic. At minimum: fraud.score_transaction, sanctions.lookup_watchlist, insights.aggregate_120d, ledger.settle. Add semantic attributes (db.system, http.route) and namespaced business ones (ledgerline.account.id, ledgerline.risk.score, ledgerline.tx.amount_minor).
4. Propagate across the Redis Stream by hand. Producer: create a PRODUCER span, inject trace context into the message fields. Consumers (notifier, audit): extract, and start a CONSUMER span with the extracted context as parent.
5. Record exceptions everywhere you currently logger.exception(...).
6. Delete the manual X-Request-Id plumbing. Trace ID replaces it and does strictly more.
7. Delete tests/test_no_observability.py. It has done its job — it guarded the "before" state so you could measure against it, and from here it's actively wrong. Everything else in tests/ must still pass: instrumenting the system is not allowed to change its behaviour, and if make test goes red after adding spans, you've broken something real. That is exactly the safety net you built it for.

Done when

• One request produces one connected trace covering all seven services, including both async consumers.
• With chaos-nightmare on, you can open a slow trace and point at the span that ate the time.
• You can explain why blocking_event_loop looks different in a trace than slow_sanctions does. (Hint: one shows a long span, the other shows a gap before spans start, plus every concurrent request in that process degrading together.)