LedgerLine/k8s

Observability platform on k3s (falken)

The Grafana stack that Ledgerline reports to. It is a port of the mentor's Ansible role (ansible/roles/observability/templates/grafana-stack/) to Kubernetes: same image versions, same configs, same pipeline behaviour. The ansible/ directory stays as the reference it was ported from.

Ledgerline itself still runs on czy with Docker Compose (see docs/roadmap/phase-1.md). Only the platform lives here.

What's deployed

Namespace ledgerline-observability. Service names match the compose service names, so every in-config URL (http://prometheus:9090/api/v1/write, tempo:4317, …) is unchanged from the Ansible templates.

Component	Image	Endpoint	Auth
Grafana	grafana/grafana:12.1.1	https://grafana.vione.cloud	Grafana login
Prometheus	prom/prometheus:v3.5.0	https://prometheus.vione.cloud	basic auth
Alertmanager	prom/alertmanager:v0.28.1	https://alertmanager.vione.cloud	basic auth
Loki (query + push)	grafana/loki:3.5.3	https://loki.vione.cloud	basic auth
Tempo (query)	grafana/tempo:2.8.2	https://tempo.vione.cloud	basic auth
Alloy UI	grafana/alloy:v1.10.2	https://alloy.vione.cloud	basic auth
OTLP/HTTP ingest (Alloy :4318)	—	https://otlp.vione.cloud	basic auth
node-exporter (falken host)	prom/node-exporter:v1.9.1	cluster-internal	—

OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.

Differences from the Ansible stack, and why

• Alloy's Docker-log half runs on czy, not here. k3s uses containerd, so there's no Docker socket, and Ledgerline's containers are on czy anyway. The discovery.docker → loki.source.docker chain is in observability/alloy-agent/config.alloy, run by the alloy-agent service in docker-compose.traefik.yml. It pushes to loki.vione.cloud. The in-cluster Alloy keeps the OTLP receiver exactly as it was: traces output only.
• Endpoints are subdomains behind Traefik + basic auth instead of raw host ports. vione.cloud sits behind Cloudflare's proxy, which only forwards 80/443 (plus a few others), so raw ports like :3000 would never work.
• Resource requests/limits on every container (falken is shared).
• App scrape job added: ledgerline (the role's prometheus_extra_scrape_jobs is empty). Prometheus scrapes each czy service at https://ledgerline.vione.cloud/metrics/<service>, which czy's Traefik routes to that container's :9464 behind basic auth. The scrape credentials are the prometheus-scrape-auth secret, mounted at /etc/prometheus/secrets.
• Tempo labels exemplars trace_id (trace_id_label_name in the overrides). Tempo's default is traceID, which the Prometheus datasource's exemplarTraceIdDestinations (trace_id) doesn't match, so span-metric exemplars rendered in Grafana but didn't link to the trace.

The CLAUDE.md platform facts still hold:

• OTLP metrics and logs are rejected: Alloy returns a 404 on /v1/metrics and /v1/logs, so the only sign is an export error in the app's own logs.
• Span metrics and the service graph come from Tempo's metrics-generator.
• The Loki derived field matches lowercase trace_id.
• Datasources are provisioned; dashboards are not.

Deploy

# once: generate credentials (k8s/.credentials, gitignored) and create secrets.
# Also writes .env.observability and observability/traefik/metrics-users for czy.
./k8s/create-secrets.sh

# deploy, or re-apply after editing any config file
kubectl apply -k k8s/

Every config file goes through a configMapGenerator. Its name carries a content hash, so editing e.g. alloy/config.alloy and re-applying rolls that pod automatically. That's the equivalent of re-running the playbook.

If the API server isn't reachable from your machine, build locally and apply on the node:

KUBECTL="ssh falken kubectl" ./k8s/create-secrets.sh
kubectl kustomize k8s/ | ssh falken kubectl apply -f -

czy files (once, then ./deploy.sh keeps it running). Copy metrics-users before the first deploy that mounts it. Otherwise Docker creates a directory at that path, and Traefik's metrics routes fail until you remove it:

scp .env.observability czy:/opt/apps/LedgerLine/.env.observability
ssh czy mkdir -p /opt/apps/LedgerLine/observability/traefik
scp observability/traefik/metrics-users czy:/opt/apps/LedgerLine/observability/traefik/metrics-users
./deploy.sh

Teardown / rebuild

kubectl delete -k k8s/      # removes everything, including PVCs (all data)

grafana-admin, obs-basic-auth and prometheus-scrape-auth are deleted with the namespace. Run create-secrets.sh again before re-applying; it reuses k8s/.credentials.

Credentials

k8s/.credentials (mode 600, gitignored):

• OBS_USER / OBS_PASSWORD: basic auth for every endpoint except Grafana.
• GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD: the Grafana login.

source k8s/.credentials
curl -s -u "$OBS_USER:$OBS_PASSWORD" https://prometheus.vione.cloud/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'

For an OpenTelemetry SDK (the header value must be URL-encoded, so the space becomes %20):

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20$(printf '%s:%s' "$OBS_USER" "$OBS_PASSWORD" | base64)"

Known gaps

• czy host metrics: node-exporter here watches falken. czy's own saturation (1 vCPU / 1 GB) is invisible until a node-exporter runs there and has a scrape path.