Add deployment phase and Traefik compose override
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b2112e55e1
commit
5e514ecf25
+71
-98
@@ -1,116 +1,89 @@
|
||||
# Phase 2 — Distributed tracing
|
||||
# Phase 2 — The model, and one signal end to end
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
|
||||
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
|
||||
described above. Waiting to begin.
|
||||
|
||||
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
|
||||
**Time-box:** one evening.
|
||||
|
||||
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
|
||||
(Resource, providers, exporters, Collector wiring) is shared across all three
|
||||
signals — do it once here and logs and metrics become incremental. Second, tracing
|
||||
is what Babak actually asked for ("trace a single request, see where it stuck"),
|
||||
and once trace IDs exist, log correlation in Phase 3 works on the first try instead
|
||||
of needing rework.
|
||||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||||
than abstract.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Trace / span / span context.** A trace is one request's journey; a span is one
|
||||
timed operation within it. Each span carries a trace ID, its own span ID, and its
|
||||
parent's span ID. That parent pointer is the entire data structure.
|
||||
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
|
||||
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
|
||||
these right is what makes the service graph render correctly.
|
||||
- **Context propagation.**
|
||||
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
|
||||
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
|
||||
carry the context explicitly.
|
||||
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
|
||||
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
|
||||
- The classic bug: any boundary that doesn't carry those headers **starts a brand
|
||||
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
|
||||
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
|
||||
producer and `.extract(...)` on the consumer, then starting the consumer span
|
||||
with that context as parent. This is the single most valuable exercise in the
|
||||
phase — most people never do it manually and never really understand it.
|
||||
- Related: `links` (for a consumer span that batches messages from many traces)
|
||||
and **Baggage** (arbitrary key/values propagated alongside trace context —
|
||||
useful, and a foot-gun, because it's sent on every hop and is visible to
|
||||
downstream systems; never put secrets in it).
|
||||
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
|
||||
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
|
||||
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
|
||||
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
|
||||
that show *which service* was slow but not *why*.
|
||||
- **Span attributes vs metric labels.** Spans are the right home for
|
||||
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
|
||||
with volume, not with distinct values. This is why "put it on the span, not the
|
||||
metric" is the standing rule.
|
||||
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
|
||||
That's how the exception ends up attached to the trace and how the `spanmetrics`
|
||||
connector in Phase 4 will count your error rate.
|
||||
- **Sampling — the concept now, the config later.**
|
||||
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
|
||||
errors at the same rate as successes.
|
||||
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
|
||||
traces, but must run in the Collector and buffer spans in memory.
|
||||
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
|
||||
now; you'll configure tail sampling in the Collector in Phase 5.
|
||||
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
|
||||
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
|
||||
unilaterally drop half of a trace its caller decided to keep.
|
||||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||||
Observability is a property of the system: can you ask a *new* question about
|
||||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||||
attributes), not just pre-aggregated counters.
|
||||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||||
- The **API** is what your code and third-party instrumentation libraries call.
|
||||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||||
library can be instrumented without forcing a dependency on any implementation.
|
||||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||||
Application owners configure the SDK; library authors only touch the API.
|
||||
- The **Collector** is a standalone process that receives, processes and exports
|
||||
telemetry. Optional, but you should use one.
|
||||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||||
that process. Attributes describe the individual span/metric/log.
|
||||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||||
aware of, not learning yet.)
|
||||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||||
seven services. The Collector also does the things you don't want in your app:
|
||||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||||
destinations.
|
||||
|
||||
### Resources
|
||||
|
||||
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
|
||||
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
|
||||
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
|
||||
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
|
||||
practical FastAPI-specific setup, including batch/queue env vars.
|
||||
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
|
||||
reading.
|
||||
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
|
||||
side by side. Look specifically at how they mix auto-instrumentation with
|
||||
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
|
||||
This is the "professional implementation" you asked to study.
|
||||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||||
it — just see what "done" looks like.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
|
||||
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
|
||||
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
|
||||
service calls it once at startup. Keep it under 60 lines.
|
||||
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
|
||||
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
|
||||
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
|
||||
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
|
||||
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
|
||||
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
|
||||
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
|
||||
`ledgerline.tx.amount_minor`).
|
||||
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
|
||||
inject trace context into the message fields. Consumers (`notifier`, `audit`):
|
||||
extract, and start a `CONSUMER` span with the extracted context as parent.
|
||||
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
|
||||
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
|
||||
more.
|
||||
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
|
||||
the "before" state so you could measure against it, and from here it's
|
||||
actively wrong. Everything else in `tests/` must still pass: instrumenting
|
||||
the system is not allowed to change its behaviour, and if `make test` goes red
|
||||
after adding spans, you've broken something real. That is exactly the safety
|
||||
net you built it for.
|
||||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||||
`ansible-galaxy collection install -r requirements.yml`, change
|
||||
`grafana_admin_password` (vault it), then
|
||||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||||
a single container with everything pre-wired. Useful for a first trace, but
|
||||
don't build on it — its pipeline differs from your real platform in exactly the
|
||||
ways that matter.)*
|
||||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||||
auto-instrumentation:
|
||||
```
|
||||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||||
opentelemetry-bootstrap -a install
|
||||
```
|
||||
and change its command to
|
||||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||||
with `OTEL_SERVICE_NAME=sanctions`,
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||||
Tempo directly.
|
||||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||||
|
||||
### Done when
|
||||
|
||||
- One request produces **one connected trace** covering all seven services,
|
||||
including both async consumers.
|
||||
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
|
||||
ate the time.
|
||||
- You can explain why `blocking_event_loop` looks different in a trace than
|
||||
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
|
||||
before spans start, plus every concurrent request in that process degrading
|
||||
together.)
|
||||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||||
loud what the Resource attributes on it mean and where they came from.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user