Add project documentation and Claude instructions

Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.

docs/roadmap/phase-2.md

@@ -0,0 +1,116 @@
+# Phase 2 — Distributed tracing
+
+*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
+
+**Status:** Not started
+
+**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
+
+**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
+
+**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
+(Resource, providers, exporters, Collector wiring) is shared across all three
+signals — do it once here and logs and metrics become incremental. Second, tracing
+is what Babak actually asked for ("trace a single request, see where it stuck"),
+and once trace IDs exist, log correlation in Phase 3 works on the first try instead
+of needing rework.
+
+### Concepts
+
+- **Trace / span / span context.** A trace is one request's journey; a span is one
+  timed operation within it. Each span carries a trace ID, its own span ID, and its
+  parent's span ID. That parent pointer is the entire data structure.
+- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
+  outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
+  these right is what makes the service graph render correctly.
+- **Context propagation.**
+  - *In-process*: OTel uses Python `contextvars`. This works across `await` — but
+    **not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
+    carry the context explicitly.
+  - *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
+    `tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
+  - The classic bug: any boundary that doesn't carry those headers **starts a brand
+    new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
+    `TraceContextTextMapPropagator().inject(...)` into the message payload on the
+    producer and `.extract(...)` on the consumer, then starting the consumer span
+    with that context as parent. This is the single most valuable exercise in the
+    phase — most people never do it manually and never really understand it.
+  - Related: `links` (for a consumer span that batches messages from many traces)
+    and **Baggage** (arbitrary key/values propagated alongside trace context —
+    useful, and a foot-gun, because it's sent on every hop and is visible to
+    downstream systems; never put secrets in it).
+- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
+  client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
+  gets you the business semantics: `score_transaction`, `check_sanctions_list`,
+  `settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
+  that show *which service* was slow but not *why*.
+- **Span attributes vs metric labels.** Spans are the right home for
+  high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
+  with volume, not with distinct values. This is why "put it on the span, not the
+  metric" is the standing rule.
+- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
+  That's how the exception ends up attached to the trace and how the `spanmetrics`
+  connector in Phase 4 will count your error rate.
+- **Sampling — the concept now, the config later.**
+  - *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
+    errors at the same rate as successes.
+  - *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
+    traces, but must run in the Collector and buffer spans in memory.
+  - For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
+    now; you'll configure tail sampling in the Collector in Phase 5.
+  - Whatever you choose, the sampling decision must **propagate** — the sampled flag
+    rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
+    unilaterally drop half of a trace its caller decided to keep.
+
+### Resources
+
+- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
+  min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
+- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
+- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
+  practical FastAPI-specific setup, including batch/queue env vars.
+- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
+  reading.
+- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
+  side by side. Look specifically at how they mix auto-instrumentation with
+  hand-rolled business spans, and how the Kafka producer/consumer propagate context.
+  This is the "professional implementation" you asked to study.
+
+### Build
+
+1. **Shared bootstrap.** Create `common/telemetry.py` with a single
+   `setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
+   `BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
+   service calls it once at startup. Keep it under 60 lines.
+2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
+   `HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
+   you now get a connected trace across `gateway → ledger → fraud → sanctions`.
+3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
+   `sanctions.lookup_watchlist`, `insights.aggregate_120d`,
+   `ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
+   namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
+   `ledgerline.tx.amount_minor`).
+4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
+   inject trace context into the message fields. Consumers (`notifier`, `audit`):
+   extract, and start a `CONSUMER` span with the extracted context as parent.
+5. **Record exceptions** everywhere you currently `logger.exception(...)`.
+6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
+   more.
+7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
+   the "before" state so you could measure against it, and from here it's
+   actively wrong. Everything else in `tests/` must still pass: instrumenting
+   the system is not allowed to change its behaviour, and if `make test` goes red
+   after adding spans, you've broken something real. That is exactly the safety
+   net you built it for.
+
+### Done when
+
+- One request produces **one connected trace** covering all seven services,
+  including both async consumers.
+- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
+  ate the time.
+- You can explain why `blocking_event_loop` looks different in a trace than
+  `slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
+  before spans start, plus every concurrent request in that process degrading
+  together.)
+