Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/roadmap/phase-6.md

@@ -45,22 +45,27 @@ recreate in seconds.*
 
 Your platform provisions **datasources only** — there is no dashboards provider at
 all. So this phase is entirely yours, and the right home for it is inside the
-Ansible role, using the same mechanism as everything else.
+platform's own manifests (`k8s/grafana/`, the k3s port of the Ansible role),
+using the same mechanism as everything else.
 
-1. **Extend the role.** Add
-   `roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
-   (a file-based dashboard provider pointing at a mounted directory), a
-   `config/grafana/dashboards/` directory for the JSON, and the tasks to template
-   and copy them. The Compose template already mounts
-   `./config/grafana/provisioning` read-only, so the provider is picked up on
-   restart.
+1. **Extend `k8s/grafana/`.** Add a file-based dashboard provider (a
+   `provisioning/dashboards/` YAML pointing at a mounted directory) and the
+   dashboard JSON. Ship both through `configMapGenerator`, exactly as
+   `datasources.yml` is shipped today, and mount them in `deployment.yaml`.
+   Two k8s-specific things to work out:
+   - A ConfigMap is capped at about 1 MiB, and client-side `kubectl apply`
+     also stores a copy in an annotation. Large generated dashboards push
+     against that.
+   - The configMap hash suffix rolls the Grafana pod on every change.
+     Decide whether that's what you want.
 2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
    Foundation SDK (Python)**: one function producing a RED dashboard for a given
    service name, looped over the seven services; a second producing a system
    overview (service graph, stream lag, pool saturation, error-budget burn). Commit
-   the generator *and* its output; the Ansible role ships the output.
-3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
-   Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
+   the generator *and* its output; `k8s/grafana/` ships the output.
+3. **The demo is now `kubectl apply -k k8s/`.** Delete every dashboard through
+   the Grafana UI, re-apply (or go further: `kubectl delete -k k8s/` and rebuild
+   the whole platform), and watch them all come back.
    That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
    than a local Makefile target, because it's the same command that builds the
    platform.
@@ -71,6 +76,6 @@ Ansible role, using the same mechanism as everything else.
    wouldn't be.
 5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.
 
-**Done when:** re-running the playbook restores every dashboard from scratch, and no
+**Done when:** `kubectl apply -k k8s/` restores every dashboard from scratch, and no
 dashboard has ever been saved through the Grafana UI.