Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

k8s/README.md

@@ -0,0 +1,117 @@
+# Observability platform on k3s (`falken`)
+
+The Grafana stack that Ledgerline reports to. It is a **port of the mentor's
+Ansible role** (`ansible/roles/observability/templates/grafana-stack/`) to
+Kubernetes: same image versions, same configs, same pipeline behaviour. The
+`ansible/` directory stays as the reference it was ported from.
+
+Ledgerline itself still runs on `czy` with Docker Compose (see
+`docs/roadmap/phase-1.md`). Only the platform lives here.
+
+## What's deployed
+
+Namespace `ledgerline-observability`. Service names match the compose service
+names, so every in-config URL (`http://prometheus:9090/api/v1/write`,
+`tempo:4317`, …) is unchanged from the Ansible templates.
+
+| Component | Image | Endpoint | Auth |
+|---|---|---|---|
+| Grafana | `grafana/grafana:12.1.1` | https://grafana.vione.cloud | Grafana login |
+| Prometheus | `prom/prometheus:v3.5.0` | https://prometheus.vione.cloud | basic auth |
+| Alertmanager | `prom/alertmanager:v0.28.1` | https://alertmanager.vione.cloud | basic auth |
+| Loki (query + push) | `grafana/loki:3.5.3` | https://loki.vione.cloud | basic auth |
+| Tempo (query) | `grafana/tempo:2.8.2` | https://tempo.vione.cloud | basic auth |
+| Alloy UI | `grafana/alloy:v1.10.2` | https://alloy.vione.cloud | basic auth |
+| OTLP/HTTP ingest (Alloy :4318) | — | https://otlp.vione.cloud | basic auth |
+| node-exporter (falken host) | `prom/node-exporter:v1.9.1` | cluster-internal | — |
+
+OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.
+
+### Differences from the Ansible stack, and why
+
+- **Alloy's Docker-log half runs on czy, not here.** k3s uses containerd, so
+  there's no Docker socket, and Ledgerline's containers are on czy anyway. The
+  `discovery.docker` → `loki.source.docker` chain is in
+  `observability/alloy-agent/config.alloy`, run by the `alloy-agent` service in
+  `docker-compose.traefik.yml`. It pushes to `loki.vione.cloud`. The in-cluster
+  Alloy keeps the OTLP receiver exactly as it was: **traces output only**.
+- **Endpoints are subdomains behind Traefik + basic auth** instead of raw host
+  ports. `vione.cloud` sits behind Cloudflare's proxy, which only forwards
+  80/443 (plus a few others), so raw ports like `:3000` would never work.
+- **Resource requests/limits** on every container (falken is shared).
+- **Not ported:** Prometheus app scrape jobs (`prometheus_extra_scrape_jobs` is
+  empty in the role too; see `prometheus/prometheus.yml`).
+
+The CLAUDE.md platform facts still hold:
+- OTLP metrics and logs are silently dropped.
+- Span metrics and the service graph come from Tempo's metrics-generator.
+- The Loki derived field matches lowercase `trace_id`.
+- Datasources are provisioned; dashboards are not.
+
+## Deploy
+
+```bash
+# once: generate credentials (k8s/.credentials, gitignored) and create secrets.
+# Also writes .env.observability for the czy agent.
+./k8s/create-secrets.sh
+
+# deploy, or re-apply after editing any config file
+kubectl apply -k k8s/
+```
+
+Every config file goes through a `configMapGenerator`. Its name carries a
+content hash, so editing e.g. `alloy/config.alloy` and re-applying rolls that
+pod automatically. That's the equivalent of re-running the playbook.
+
+If the API server isn't reachable from your machine, build locally and apply
+on the node:
+
+```bash
+KUBECTL="ssh falken kubectl" ./k8s/create-secrets.sh
+kubectl kustomize k8s/ | ssh falken kubectl apply -f -
+```
+
+**czy agent** (once, then `./deploy.sh` keeps it running):
+
+```bash
+scp .env.observability czy:/opt/apps/LedgerLine/.env.observability
+./deploy.sh
+```
+
+## Teardown / rebuild
+
+```bash
+kubectl delete -k k8s/      # removes everything, including PVCs (all data)
+```
+
+`grafana-admin` and `obs-basic-auth` are deleted with the namespace. Run
+`create-secrets.sh` again before re-applying; it reuses `k8s/.credentials`.
+
+## Credentials
+
+`k8s/.credentials` (mode 600, gitignored):
+- `OBS_USER` / `OBS_PASSWORD`: basic auth for every endpoint except Grafana.
+- `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD`: the Grafana login.
+
+```bash
+source k8s/.credentials
+curl -s -u "$OBS_USER:$OBS_PASSWORD" https://prometheus.vione.cloud/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'
+```
+
+For an OpenTelemetry SDK (the header value must be URL-encoded, so the space
+becomes `%20`):
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20$(printf '%s:%s' "$OBS_USER" "$OBS_PASSWORD" | base64)"
+```
+
+## Known gaps
+
+- **App metrics (Phase 5):** Prometheus here can't reach czy's services.
+  Their host ports are stripped by the Traefik override. Decide the path
+  before adding scrape jobs.
+- **czy host metrics:** node-exporter here watches falken. czy's own
+  saturation (1 vCPU / 1 GB) is invisible until a node-exporter runs there
+  and has a scrape path.