Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 13:24:38 +03:30
co-authored by Claude Opus 5.5
parent 34a11da848
commit 7719ef5fcf
57 changed files with 1507 additions and 110 deletions
+13 -7
View File
@@ -62,10 +62,14 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
and can be enabled per-tenant — at the cost of load on the trace store.
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
click a spike on a latency graph and land in the exact trace. On your platform
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
`name: trace_id`. If you emit your own histograms with exemplars, the label must
be `trace_id` to match.
this is wired: Tempo's generator sets `send_exemplars: true`, Prometheus stores
them (`--enable-feature=exemplar-storage`, added in `k8s/`; the Ansible role
lacked it), and the Prometheus datasource declares `exemplarTraceIdDestinations`
with `name: trace_id`. If you emit your own histograms with exemplars, the label
must be `trace_id` to match. **But not quite end to end:** Tempo's generator
labels its exemplars `traceID` (verified live), so the span-metric exemplars
don't match that destination. Find the fix yourself and choose one: the
datasource, Tempo, or both.
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
@@ -92,9 +96,11 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
metrics, so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add them to
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
playbook. Prometheus scrapes your services directly. Note the irony and the
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
**First decide how Prometheus on falken reaches czy**: the Traefik
override strips every service's host port (see `phase-1.md`). Note the irony and the
lesson — you're back to a pull model and a second client library, which is
exactly the fragmentation OTel exists to fix.
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to