Movses AslanyanandClaude Opus 5.5
7719ef5fcf
k8s/ is now a faithful port of the mentor's Ansible Grafana stack (Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter) for the ledgerline-observability namespace on falken: same versions and configs, one *.vione.cloud ingress per tool behind Traefik basic auth, secrets generated by create-secrets.sh into gitignored files. Two fixes over the Ansible role, both verified live: Tempo's metrics-generator needs traces_storage or it runs no processors, and Prometheus needs exemplar-storage to keep exemplars. Ledgerline stays on czy; an Alloy agent in the Traefik override ships its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
129 lines
6.5 KiB
Markdown
129 lines
6.5 KiB
Markdown
# Phase 2 — The model, and one signal end to end
|
||
|
||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||
|
||
**Status:** Not started
|
||
|
||
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
|
||
described above. Waiting to begin.
|
||
|
||
**Time-box:** one evening.
|
||
|
||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||
than abstract.
|
||
|
||
### Concepts
|
||
|
||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||
Observability is a property of the system: can you ask a *new* question about
|
||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||
attributes), not just pre-aggregated counters.
|
||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||
- The **API** is what your code and third-party instrumentation libraries call.
|
||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||
library can be instrumented without forcing a dependency on any implementation.
|
||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||
Application owners configure the SDK; library authors only touch the API.
|
||
- The **Collector** is a standalone process that receives, processes and exports
|
||
telemetry. Optional, but you should use one.
|
||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||
that process. Attributes describe the individual span/metric/log.
|
||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||
aware of, not learning yet.)
|
||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||
seven services. The Collector also does the things you don't want in your app:
|
||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||
destinations.
|
||
|
||
### Resources
|
||
|
||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||
it — just see what "done" looks like.
|
||
|
||
### Build
|
||
|
||
1. **Deploy the platform.** *Deployed 2026-10-12.* The mentor's Ansible
|
||
Grafana stack was ported to the k3s cluster `falken` as `k8s/`, keeping the
|
||
same versions, configs and pipeline. Deploy with `kubectl apply -k k8s/`.
|
||
`k8s/README.md` covers deploy/teardown, credentials and every deviation
|
||
from the Ansible role.
|
||
|
||
**Why not the Ansible route on a VM:**
|
||
- `czy` (1 GB) can't host it. The platform README recommends 2 vCPU / 4 GB.
|
||
- `*.vione.cloud` sits behind Cloudflare's proxy, which only forwards a few
|
||
ports (80/443, 8080, 8443, …), so raw ports like `:3000` never reach any
|
||
origin. That's why `ledgerline.vione.cloud:3000` showed nothing.
|
||
|
||
**Endpoints:** each one is a subdomain behind Traefik. Everything except
|
||
Grafana uses basic auth (`k8s/.credentials`).
|
||
|
||
| | |
|
||
|---|---|
|
||
| Grafana | https://grafana.vione.cloud |
|
||
| Prometheus | https://prometheus.vione.cloud |
|
||
| Alertmanager | https://alertmanager.vione.cloud |
|
||
| Loki | https://loki.vione.cloud |
|
||
| Tempo (query) | https://tempo.vione.cloud |
|
||
| Alloy UI | https://alloy.vione.cloud |
|
||
| OTLP/HTTP ingest | https://otlp.vione.cloud |
|
||
|
||
Container logs from `czy` reach Loki through an Alloy agent there (the
|
||
`alloy-agent` service in `docker-compose.traefik.yml`).
|
||
|
||
**Verified live:** all of the following worked, using a scratch trace sent
|
||
from `czy`.
|
||
- Every endpoint answers, and returns 401 without auth.
|
||
- Prometheus scrapes all six platform targets.
|
||
- Traces flow czy → Alloy → Tempo.
|
||
- Tempo's metrics-generator emits `traces_spanmetrics_*` and
|
||
`traces_service_graph_*` with exemplars.
|
||
- czy container logs land in Loki.
|
||
|
||
**Two platform bugs, fixed in the port.** As shipped, the Ansible role
|
||
produced no span metrics or service graph (Tempo needs
|
||
`metrics_generator.traces_storage`), and Prometheus discarded exemplars
|
||
(missing `--enable-feature=exemplar-storage`).
|
||
|
||
**One issue left open on purpose:** Tempo labels its exemplars `traceID`,
|
||
but the datasource's `exemplarTraceIdDestinations` expects `trace_id`.
|
||
|
||
Now open the Alloy UI and **read the component graph**. It shows you the
|
||
exact pipeline, including the fact that the OTLP receiver has only a traces
|
||
output. Seeing that in the UI before you get bitten by it is worth ten
|
||
minutes.
|
||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||
auto-instrumentation:
|
||
```
|
||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||
opentelemetry-bootstrap -a install
|
||
```
|
||
and change its command to
|
||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||
with `OTEL_SERVICE_NAME=sanctions`,
|
||
`OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud`,
|
||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||
Tempo directly — plus the basic-auth header
|
||
`OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 user:pass>`
|
||
(URL-encoded: the space is `%20`; see `k8s/README.md`). Keep the
|
||
credential in the gitignored `.env.observability` on czy, not in compose.
|
||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||
|
||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||
loud what the Resource attributes on it mean and where they came from.
|
||
|