Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 13:24:38 +03:30
co-authored by Claude Opus 5.5
parent 34a11da848
commit 7719ef5fcf
57 changed files with 1507 additions and 110 deletions
+21
View File
@@ -59,5 +59,26 @@ services:
notifier:
ports: !reset []
# Ships every container's stdout/stderr to the platform's Loki on falken
# (k8s/), labelled `container`. Credentials come from .env.observability,
# generated by k8s/create-secrets.sh and copied to the server once.
# Publishes no ports; memory-capped because czy has 1 GB of RAM. Steady
# state is ~70 MiB, but the initial backfill of every container's log
# history peaked at ~150 MiB in testing, hence the headroom.
alloy-agent:
image: grafana/alloy:v1.10.2
restart: unless-stopped
command:
- run
- --storage.path=/var/lib/alloy/data
- /etc/alloy/config.alloy
env_file: .env.observability
mem_limit: 256m
volumes:
- ./observability/alloy-agent/config.alloy:/etc/alloy/config.alloy:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- alloy-agent-data:/var/lib/alloy/data
volumes:
letsencrypt:
alloy-agent-data: