Movses AslanyanandClaude Opus 5.5
34a11da848
phase-1: deploy via deploy.sh, correct the container count, and note that stripped host ports leave no path for the Phase 5 scrape. phase-2: explain why Grafana is not on the Ledgerline server and why Cloudflare's proxy never forwards :3000 or the other platform ports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
109 lines
6.0 KiB
Markdown
109 lines
6.0 KiB
Markdown
# Phase 2 — The model, and one signal end to end
|
||
|
||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||
|
||
**Status:** Not started
|
||
|
||
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
|
||
described above. Waiting to begin.
|
||
|
||
**Time-box:** one evening.
|
||
|
||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||
than abstract.
|
||
|
||
### Concepts
|
||
|
||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||
Observability is a property of the system: can you ask a *new* question about
|
||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||
attributes), not just pre-aggregated counters.
|
||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||
- The **API** is what your code and third-party instrumentation libraries call.
|
||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||
library can be instrumented without forcing a dependency on any implementation.
|
||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||
Application owners configure the SDK; library authors only touch the API.
|
||
- The **Collector** is a standalone process that receives, processes and exports
|
||
telemetry. Optional, but you should use one.
|
||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||
that process. Attributes describe the individual span/metric/log.
|
||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||
aware of, not learning yet.)
|
||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||
seven services. The Collector also does the things you don't want in your app:
|
||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||
destinations.
|
||
|
||
### Resources
|
||
|
||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||
it — just see what "done" looks like.
|
||
|
||
### Build
|
||
|
||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||
`ansible-galaxy collection install -r requirements.yml`, change
|
||
`grafana_admin_password` (vault it), then
|
||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||
|
||
**Where the platform lives — not on the Ledgerline server.** Grafana (3000)
|
||
and the rest come from the Ansible stack, not from Ledgerline's compose
|
||
files, so `ledgerline.vione.cloud:3000` will never show Grafana. Three
|
||
reasons it belongs on its own host:
|
||
- `czy` has 1 GB RAM and Ledgerline already fills it; the platform README
|
||
recommends 2 vCPU / 4 GB for the Grafana stack.
|
||
- Phase 1's rule is that only 22/80/443 are open on `czy`; the platform
|
||
publishes seven more ports.
|
||
- `ledgerline.vione.cloud` is behind Cloudflare's proxy, which only forwards
|
||
ports 80, 8080, 8880, 2052, 2082, 2086, 2095 (HTTP) and 443, 2053, 2083,
|
||
2087, 2096, 8443 (HTTPS). Port 3000 (and 3100, 3200, 4318, 9090, 12345)
|
||
never reaches the origin through a proxied hostname.
|
||
|
||
So reach the platform by the obs host's IP, an SSH tunnel, or a separate
|
||
**DNS-only** (grey-cloud) record. The platform's ports are published on all
|
||
interfaces with little or no auth — firewall them to your own IP plus
|
||
`czy`'s IP (which needs 4318 for OTLP). `<obs-host>` below means that host,
|
||
reached from `czy` over the internet.
|
||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||
a single container with everything pre-wired. Useful for a first trace, but
|
||
don't build on it — its pipeline differs from your real platform in exactly the
|
||
ways that matter.)*
|
||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||
auto-instrumentation:
|
||
```
|
||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||
opentelemetry-bootstrap -a install
|
||
```
|
||
and change its command to
|
||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||
with `OTEL_SERVICE_NAME=sanctions`,
|
||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||
Tempo directly.
|
||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||
|
||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||
loud what the Resource attributes on it mean and where they came from.
|
||
|