Port the observability platform to k3s and ship czy logs to it

k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

k8s/create-secrets.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Create the platform's secrets in the cluster. Idempotent: credentials are
+# generated once into k8s/.credentials (gitignored, mode 600) and reused on
+# every later run, so re-running only re-applies the same secrets.
+#
+#   ./k8s/create-secrets.sh
+#
+# Creates, in namespace ledgerline-observability:
+#   obs-basic-auth  htpasswd for the Traefik basic-auth middleware
+#                   (Prometheus, Alertmanager, Loki, Tempo, Alloy UI, OTLP)
+#   grafana-admin   Grafana admin user/password
+# and writes .env.observability at the repo root (gitignored) for the czy-side
+# Alloy log agent; copy it to the server once (see k8s/README.md).
+#
+# KUBECTL defaults to `kubectl`; e.g. KUBECTL="ssh falken kubectl" to run it on
+# the node when the API server isn't reachable from here.
+set -euo pipefail
+
+cd "$(dirname "$0")"
+KUBECTL="${KUBECTL:-kubectl}"
+NS=ledgerline-observability
+CREDS=.credentials
+
+if [ ! -f "$CREDS" ]; then
+  umask 077
+  cat > "$CREDS" <<EOF
+OBS_USER=ledgerline
+OBS_PASSWORD=$(openssl rand -hex 24)
+GRAFANA_ADMIN_USER=admin
+GRAFANA_ADMIN_PASSWORD=$(openssl rand -hex 24)
+EOF
+  echo "generated $PWD/$CREDS"
+fi
+# shellcheck disable=SC1090
+source "$CREDS"
+
+# Secrets are rendered locally (no API access needed) and applied through
+# $KUBECTL, so they never appear on a command line on the remote side.
+$KUBECTL apply -f - < namespace.yaml
+
+kubectl create secret generic obs-basic-auth -n "$NS" \
+  --from-literal=users="$(htpasswd -nbB "$OBS_USER" "$OBS_PASSWORD")" \
+  --dry-run=client -o yaml | $KUBECTL apply -f -
+
+kubectl create secret generic grafana-admin -n "$NS" \
+  --from-literal=admin-user="$GRAFANA_ADMIN_USER" \
+  --from-literal=admin-password="$GRAFANA_ADMIN_PASSWORD" \
+  --dry-run=client -o yaml | $KUBECTL apply -f -
+
+(
+  umask 077
+  cat > ../.env.observability <<EOF
+# Credentials for the Alloy log agent on czy (docker-compose.traefik.yml).
+# Generated by k8s/create-secrets.sh -- do not commit.
+LOKI_PUSH_URL=https://loki.vione.cloud/loki/api/v1/push
+LOKI_PUSH_USER=$OBS_USER
+LOKI_PUSH_PASSWORD=$OBS_PASSWORD
+EOF
+)
+echo "wrote $(cd .. && pwd)/.env.observability"