90 lines
4.9 KiB
Markdown
90 lines
4.9 KiB
Markdown
|
# Phase 1 — The model, and one signal end to end
|
|||
|
|
|
|||
|
|
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
|
|||
|
|
|
|||
|
|
**Status:** Not started
|
|||
|
|
|
|||
|
|
**Message:** Blocked on nothing — Phase 0 is done and the platform is
|
|||
|
|
described above. Waiting to begin.
|
|||
|
|
|
|||
|
|
**Time-box:** one evening.
|
|||
|
|
|
|||
|
|
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
|||
|
|
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
|||
|
|
than abstract.
|
|||
|
|
|
|||
|
|
### Concepts
|
|||
|
|
|
|||
|
|
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
|||
|
|
(known unknowns): dashboards and alerts built from predefined aggregates.
|
|||
|
|
Observability is a property of the system: can you ask a *new* question about
|
|||
|
|
production, that you never anticipated, without shipping new code? Charity Majors'
|
|||
|
|
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
|||
|
|
practical implication for you: emit **wide, high-dimensional events** (rich span
|
|||
|
|
attributes), not just pre-aggregated counters.
|
|||
|
|
- **The OTel architecture split: API vs SDK vs Collector.**
|
|||
|
|
- The **API** is what your code and third-party instrumentation libraries call.
|
|||
|
|
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
|||
|
|
library can be instrumented without forcing a dependency on any implementation.
|
|||
|
|
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
|||
|
|
Application owners configure the SDK; library authors only touch the API.
|
|||
|
|
- The **Collector** is a standalone process that receives, processes and exports
|
|||
|
|
telemetry. Optional, but you should use one.
|
|||
|
|
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
|||
|
|
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
|||
|
|
`service.version`, `deployment.environment`) and is attached to every signal from
|
|||
|
|
that process. Attributes describe the individual span/metric/log.
|
|||
|
|
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
|||
|
|
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
|||
|
|
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
|||
|
|
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
|||
|
|
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
|||
|
|
aware of, not learning yet.)
|
|||
|
|
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
|||
|
|
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
|||
|
|
seven services. The Collector also does the things you don't want in your app:
|
|||
|
|
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
|||
|
|
destinations.
|
|||
|
|
|
|||
|
|
### Resources
|
|||
|
|
|
|||
|
|
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
|||
|
|
M.E.L.T. and context-propagation sections; skip the Node code.
|
|||
|
|
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
|||
|
|
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
|||
|
|
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
|||
|
|
it — just see what "done" looks like.
|
|||
|
|
|
|||
|
|
### Build
|
|||
|
|
|
|||
|
|
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
|||
|
|
`ansible-galaxy collection install -r requirements.yml`, change
|
|||
|
|
`grafana_admin_password` (vault it), then
|
|||
|
|
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
|||
|
|
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
|||
|
|
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
|||
|
|
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
|||
|
|
pipeline, including the fact that the OTLP receiver has only a traces output.
|
|||
|
|
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
|||
|
|
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
|||
|
|
a single container with everything pre-wired. Useful for a first trace, but
|
|||
|
|
don't build on it — its pipeline differs from your real platform in exactly the
|
|||
|
|
ways that matter.)*
|
|||
|
|
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
|||
|
|
auto-instrumentation:
|
|||
|
|
```
|
|||
|
|
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
|||
|
|
opentelemetry-bootstrap -a install
|
|||
|
|
```
|
|||
|
|
and change its command to
|
|||
|
|
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
|||
|
|
with `OTEL_SERVICE_NAME=sanctions`,
|
|||
|
|
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
|||
|
|
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
|||
|
|
Tempo directly.
|
|||
|
|
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
|||
|
|
|
|||
|
|
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
|||
|
|
loud what the Resource attributes on it mean and where they came from.
|
|||
|
|
|