Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
117 lines
6.9 KiB
Markdown
117 lines
6.9 KiB
Markdown
# Phase 2 — Distributed tracing
|
||
|
||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||
|
||
**Status:** Not started
|
||
|
||
**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
|
||
|
||
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
|
||
|
||
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
|
||
(Resource, providers, exporters, Collector wiring) is shared across all three
|
||
signals — do it once here and logs and metrics become incremental. Second, tracing
|
||
is what Babak actually asked for ("trace a single request, see where it stuck"),
|
||
and once trace IDs exist, log correlation in Phase 3 works on the first try instead
|
||
of needing rework.
|
||
|
||
### Concepts
|
||
|
||
- **Trace / span / span context.** A trace is one request's journey; a span is one
|
||
timed operation within it. Each span carries a trace ID, its own span ID, and its
|
||
parent's span ID. That parent pointer is the entire data structure.
|
||
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
|
||
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
|
||
these right is what makes the service graph render correctly.
|
||
- **Context propagation.**
|
||
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
|
||
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
|
||
carry the context explicitly.
|
||
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
|
||
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
|
||
- The classic bug: any boundary that doesn't carry those headers **starts a brand
|
||
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
|
||
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
|
||
producer and `.extract(...)` on the consumer, then starting the consumer span
|
||
with that context as parent. This is the single most valuable exercise in the
|
||
phase — most people never do it manually and never really understand it.
|
||
- Related: `links` (for a consumer span that batches messages from many traces)
|
||
and **Baggage** (arbitrary key/values propagated alongside trace context —
|
||
useful, and a foot-gun, because it's sent on every hop and is visible to
|
||
downstream systems; never put secrets in it).
|
||
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
|
||
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
|
||
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
|
||
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
|
||
that show *which service* was slow but not *why*.
|
||
- **Span attributes vs metric labels.** Spans are the right home for
|
||
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
|
||
with volume, not with distinct values. This is why "put it on the span, not the
|
||
metric" is the standing rule.
|
||
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
|
||
That's how the exception ends up attached to the trace and how the `spanmetrics`
|
||
connector in Phase 4 will count your error rate.
|
||
- **Sampling — the concept now, the config later.**
|
||
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
|
||
errors at the same rate as successes.
|
||
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
|
||
traces, but must run in the Collector and buffer spans in memory.
|
||
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
|
||
now; you'll configure tail sampling in the Collector in Phase 5.
|
||
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
|
||
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
|
||
unilaterally drop half of a trace its caller decided to keep.
|
||
|
||
### Resources
|
||
|
||
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
|
||
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
|
||
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
|
||
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
|
||
practical FastAPI-specific setup, including batch/queue env vars.
|
||
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
|
||
reading.
|
||
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
|
||
side by side. Look specifically at how they mix auto-instrumentation with
|
||
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
|
||
This is the "professional implementation" you asked to study.
|
||
|
||
### Build
|
||
|
||
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
|
||
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
|
||
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
|
||
service calls it once at startup. Keep it under 60 lines.
|
||
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
|
||
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
|
||
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
|
||
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
|
||
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
|
||
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
|
||
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
|
||
`ledgerline.tx.amount_minor`).
|
||
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
|
||
inject trace context into the message fields. Consumers (`notifier`, `audit`):
|
||
extract, and start a `CONSUMER` span with the extracted context as parent.
|
||
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
|
||
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
|
||
more.
|
||
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
|
||
the "before" state so you could measure against it, and from here it's
|
||
actively wrong. Everything else in `tests/` must still pass: instrumenting
|
||
the system is not allowed to change its behaviour, and if `make test` goes red
|
||
after adding spans, you've broken something real. That is exactly the safety
|
||
net you built it for.
|
||
|
||
### Done when
|
||
|
||
- One request produces **one connected trace** covering all seven services,
|
||
including both async consumers.
|
||
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
|
||
ate the time.
|
||
- You can explain why `blocking_event_loop` looks different in a trace than
|
||
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
|
||
before spans start, plus every concurrent request in that process degrading
|
||
together.)
|
||
|