Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 13:24:38 +03:30
co-authored by Claude Opus 5.5
parent 34a11da848
commit 7719ef5fcf
57 changed files with 1507 additions and 110 deletions
+6 -5
View File
@@ -21,7 +21,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
| 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
@@ -37,8 +37,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
Traefik and a Let's Encrypt certificate, served at a real domain. Plumbing
only — no telemetry, no app changes. **Done** — see `phase-1.md`.
- **2 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
Ansible and get one auto-instrumented trace into Tempo.
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform
(the mentor's Ansible stack, ported to k3s as `k8s/` — already deployed on
`falken`) and get one auto-instrumented trace into Tempo.
- **3 — Distributed tracing.** The heart of the task: auto- and
manual-instrument all seven services, and hand-propagate trace context
across the Redis Stream hop, which carries none by default.
@@ -50,8 +51,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
silently. Tempo's metrics-generator already produces span metrics and the
service graph — don't duplicate it in the collector.
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
Grafana Foundation SDK and ship it through the Ansible role, so "destroy
and recreate in seconds" is a real, repeatable command.
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
"destroy and recreate in seconds" is a real, repeatable command.
- **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
be averaged, what coordinated omission is, and drive Ledgerline with
arrival-rate executors to get a clean before/after load profile.