Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/roadmap/phase-2.md

@@ -57,38 +57,55 @@ than abstract.
 
 ### Build
 
-1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
-   `ansible-galaxy collection install -r requirements.yml`, change
-   `grafana_admin_password` (vault it), then
-   `ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
-   README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
-   Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
+1. **Deploy the platform.** *Deployed 2026-10-12.* The mentor's Ansible
+   Grafana stack was ported to the k3s cluster `falken` as `k8s/`, keeping the
+   same versions, configs and pipeline. Deploy with `kubectl apply -k k8s/`.
+   `k8s/README.md` covers deploy/teardown, credentials and every deviation
+   from the Ansible role.
 
-   **Where the platform lives — not on the Ledgerline server.** Grafana (3000)
-   and the rest come from the Ansible stack, not from Ledgerline's compose
-   files, so `ledgerline.vione.cloud:3000` will never show Grafana. Three
-   reasons it belongs on its own host:
-   - `czy` has 1 GB RAM and Ledgerline already fills it; the platform README
-     recommends 2 vCPU / 4 GB for the Grafana stack.
-   - Phase 1's rule is that only 22/80/443 are open on `czy`; the platform
-     publishes seven more ports.
-   - `ledgerline.vione.cloud` is behind Cloudflare's proxy, which only forwards
-     ports 80, 8080, 8880, 2052, 2082, 2086, 2095 (HTTP) and 443, 2053, 2083,
-     2087, 2096, 8443 (HTTPS). Port 3000 (and 3100, 3200, 4318, 9090, 12345)
-     never reaches the origin through a proxied hostname.
+   **Why not the Ansible route on a VM:**
+   - `czy` (1 GB) can't host it. The platform README recommends 2 vCPU / 4 GB.
+   - `*.vione.cloud` sits behind Cloudflare's proxy, which only forwards a few
+     ports (80/443, 8080, 8443, …), so raw ports like `:3000` never reach any
+     origin. That's why `ledgerline.vione.cloud:3000` showed nothing.
 
-   So reach the platform by the obs host's IP, an SSH tunnel, or a separate
-   **DNS-only** (grey-cloud) record. The platform's ports are published on all
-   interfaces with little or no auth — firewall them to your own IP plus
-   `czy`'s IP (which needs 4318 for OTLP). `<obs-host>` below means that host,
-   reached from `czy` over the internet.
-   Then open the Alloy UI and **read the component graph** — it shows you the exact
-   pipeline, including the fact that the OTLP receiver has only a traces output.
-   Seeing that in the UI before you get bitten by it is worth ten minutes.
-   *(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
-   a single container with everything pre-wired. Useful for a first trace, but
-   don't build on it — its pipeline differs from your real platform in exactly the
-   ways that matter.)*
+   **Endpoints:** each one is a subdomain behind Traefik. Everything except
+   Grafana uses basic auth (`k8s/.credentials`).
+
+   | | |
+   |---|---|
+   | Grafana | https://grafana.vione.cloud |
+   | Prometheus | https://prometheus.vione.cloud |
+   | Alertmanager | https://alertmanager.vione.cloud |
+   | Loki | https://loki.vione.cloud |
+   | Tempo (query) | https://tempo.vione.cloud |
+   | Alloy UI | https://alloy.vione.cloud |
+   | OTLP/HTTP ingest | https://otlp.vione.cloud |
+
+   Container logs from `czy` reach Loki through an Alloy agent there (the
+   `alloy-agent` service in `docker-compose.traefik.yml`).
+
+   **Verified live:** all of the following worked, using a scratch trace sent
+   from `czy`.
+   - Every endpoint answers, and returns 401 without auth.
+   - Prometheus scrapes all six platform targets.
+   - Traces flow czy → Alloy → Tempo.
+   - Tempo's metrics-generator emits `traces_spanmetrics_*` and
+     `traces_service_graph_*` with exemplars.
+   - czy container logs land in Loki.
+
+   **Two platform bugs, fixed in the port.** As shipped, the Ansible role
+   produced no span metrics or service graph (Tempo needs
+   `metrics_generator.traces_storage`), and Prometheus discarded exemplars
+   (missing `--enable-feature=exemplar-storage`).
+
+   **One issue left open on purpose:** Tempo labels its exemplars `traceID`,
+   but the datasource's `exemplarTraceIdDestinations` expects `trace_id`.
+
+   Now open the Alloy UI and **read the component graph**. It shows you the
+   exact pipeline, including the fact that the OTLP receiver has only a traces
+   output. Seeing that in the UI before you get bitten by it is worth ten
+   minutes.
 2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
    auto-instrumentation:
    ```
@@ -98,9 +115,12 @@ than abstract.
    and change its command to
    `opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
    with `OTEL_SERVICE_NAME=sanctions`,
-   `OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
+   `OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud`,
    `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
-   Tempo directly.
+   Tempo directly — plus the basic-auth header
+   `OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 user:pass>`
+   (URL-encoded: the space is `%20`; see `k8s/README.md`). Keep the
+   credential in the gitignored `.env.observability` on czy, not in compose.
 3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
 
 **Done when:** you can see a single-service trace in Tempo, and you can explain out