Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 13:24:38 +03:30
co-authored by Claude Opus 5.5
parent 34a11da848
commit 7719ef5fcf
57 changed files with 1507 additions and 110 deletions
+7 -4
View File
@@ -72,17 +72,20 @@ means anything).
3. Add a redaction processor with a denylist of field names.
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
with a `container` label (Alloy's relabel rule sets it from the Docker container
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
name, e.g. `ledgerline-gateway-1`). Here that's the Alloy agent on czy
(`observability/alloy-agent/config.alloy`, already running), not the
cluster's Alloy. Verify the `trace_id` field is present *and lowercase* in
the JSON body.
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
the receiver shows traffic, and the pipeline shows no logs output. Now extend
`config.alloy.j2` — add a `logs` output on the receiver plus an
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
`k8s/alloy/config.alloy` — add a `logs` output on the receiver plus an
`otelcol.exporter.loki` → `loki.write` chain — `kubectl apply -k k8s/`, and watch
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
That is the most transferable half-hour in the whole project.
6. Confirm the provisioned correlation works in both directions: Loki's
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
(`filterByTraceID: true`) → Loki. Both are already in `k8s/grafana/datasources.yml`; you
are verifying, not writing. If a link doesn't appear, the regex didn't match
your field name — check that before anything else.
6. **Migrate the test suite's log-counting helpers.** Three pathology tests