LedgerLine/k8s

Observability platform on k3s (falken)

The Grafana stack that Ledgerline reports to. It is a port of the mentor's Ansible role (ansible/roles/observability/templates/grafana-stack/) to Kubernetes: same image versions, same configs, same pipeline behaviour. The ansible/ directory stays as the reference it was ported from.

Ledgerline itself still runs on czy with Docker Compose (see docs/roadmap/phase-1.md). Only the platform lives here.

What's deployed

Namespace ledgerline-observability. Service names match the compose service names, so every in-config URL (http://prometheus:9090/api/v1/write, tempo:4317, …) is unchanged from the Ansible templates.

Component	Image	Endpoint	Auth
Grafana	grafana/grafana:12.1.1	https://grafana.vione.cloud	Grafana login
Prometheus	prom/prometheus:v3.5.0	https://prometheus.vione.cloud	basic auth
Alertmanager	prom/alertmanager:v0.28.1	https://alertmanager.vione.cloud	basic auth
Loki (query + push)	grafana/loki:3.5.3	https://loki.vione.cloud	basic auth
Tempo (query)	grafana/tempo:2.8.2	https://tempo.vione.cloud	basic auth
Alloy UI	grafana/alloy:v1.10.2	https://alloy.vione.cloud	basic auth
OTLP/HTTP ingest (Alloy :4318)	—	https://otlp.vione.cloud	basic auth
node-exporter (falken host)	prom/node-exporter:v1.9.1	cluster-internal	—

OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.

Differences from the Ansible stack, and why

• Alloy's Docker-log half runs on czy, not here. k3s uses containerd, so there's no Docker socket, and Ledgerline's containers are on czy anyway. The discovery.docker → loki.source.docker chain is in observability/alloy-agent/config.alloy, run by the alloy-agent service in docker-compose.traefik.yml. It pushes to loki.vione.cloud. The in-cluster Alloy keeps the OTLP receiver exactly as it was: traces output only.
• Endpoints are subdomains behind Traefik + basic auth instead of raw host ports. vione.cloud sits behind Cloudflare's proxy, which only forwards 80/443 (plus a few others), so raw ports like :3000 would never work.
• Resource requests/limits on every container (falken is shared).
• Not ported: Prometheus app scrape jobs (prometheus_extra_scrape_jobs is empty in the role too; see prometheus/prometheus.yml).

The CLAUDE.md platform facts still hold:

• OTLP metrics and logs are rejected: Alloy returns a 404 on /v1/metrics and /v1/logs, so the only sign is an export error in the app's own logs.
• Span metrics and the service graph come from Tempo's metrics-generator.
• The Loki derived field matches lowercase trace_id.
• Datasources are provisioned; dashboards are not.

Deploy

# once: generate credentials (k8s/.credentials, gitignored) and create secrets.
# Also writes .env.observability for the czy agent.
./k8s/create-secrets.sh

# deploy, or re-apply after editing any config file
kubectl apply -k k8s/

Every config file goes through a configMapGenerator. Its name carries a content hash, so editing e.g. alloy/config.alloy and re-applying rolls that pod automatically. That's the equivalent of re-running the playbook.

If the API server isn't reachable from your machine, build locally and apply on the node:

KUBECTL="ssh falken kubectl" ./k8s/create-secrets.sh
kubectl kustomize k8s/ | ssh falken kubectl apply -f -

czy agent (once, then ./deploy.sh keeps it running):

scp .env.observability czy:/opt/apps/LedgerLine/.env.observability
./deploy.sh

Teardown / rebuild

kubectl delete -k k8s/      # removes everything, including PVCs (all data)

grafana-admin and obs-basic-auth are deleted with the namespace. Run create-secrets.sh again before re-applying; it reuses k8s/.credentials.

Credentials

k8s/.credentials (mode 600, gitignored):

• OBS_USER / OBS_PASSWORD: basic auth for every endpoint except Grafana.
• GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD: the Grafana login.

source k8s/.credentials
curl -s -u "$OBS_USER:$OBS_PASSWORD" https://prometheus.vione.cloud/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'

For an OpenTelemetry SDK (the header value must be URL-encoded, so the space becomes %20):

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20$(printf '%s:%s' "$OBS_USER" "$OBS_PASSWORD" | base64)"

Known gaps

• App metrics (Phase 5): Prometheus here can't reach czy's services. Their host ports are stripped by the Traefik override. Decide the path before adding scrape jobs.
• czy host metrics: node-exporter here watches falken. czy's own saturation (1 vCPU / 1 GB) is invisible until a node-exporter runs there and has a scrape path.