Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
90 lines
4.9 KiB
Markdown
90 lines
4.9 KiB
Markdown
# Phase 1 — The model, and one signal end to end
|
||
|
||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
|
||
|
||
**Status:** Not started
|
||
|
||
**Message:** Blocked on nothing — Phase 0 is done and the platform is
|
||
described above. Waiting to begin.
|
||
|
||
**Time-box:** one evening.
|
||
|
||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||
than abstract.
|
||
|
||
### Concepts
|
||
|
||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||
Observability is a property of the system: can you ask a *new* question about
|
||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||
attributes), not just pre-aggregated counters.
|
||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||
- The **API** is what your code and third-party instrumentation libraries call.
|
||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||
library can be instrumented without forcing a dependency on any implementation.
|
||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||
Application owners configure the SDK; library authors only touch the API.
|
||
- The **Collector** is a standalone process that receives, processes and exports
|
||
telemetry. Optional, but you should use one.
|
||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||
that process. Attributes describe the individual span/metric/log.
|
||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||
aware of, not learning yet.)
|
||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||
seven services. The Collector also does the things you don't want in your app:
|
||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||
destinations.
|
||
|
||
### Resources
|
||
|
||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||
it — just see what "done" looks like.
|
||
|
||
### Build
|
||
|
||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||
`ansible-galaxy collection install -r requirements.yml`, change
|
||
`grafana_admin_password` (vault it), then
|
||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||
a single container with everything pre-wired. Useful for a first trace, but
|
||
don't build on it — its pipeline differs from your real platform in exactly the
|
||
ways that matter.)*
|
||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||
auto-instrumentation:
|
||
```
|
||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||
opentelemetry-bootstrap -a install
|
||
```
|
||
and change its command to
|
||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||
with `OTEL_SERVICE_NAME=sanctions`,
|
||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||
Tempo directly.
|
||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||
|
||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||
loud what the Resource attributes on it mean and where they came from.
|
||
|