# Observability platform on k3s (`falken`)

The Grafana stack that Ledgerline reports to. It is a **port of the mentor's
Ansible role** (`ansible/roles/observability/templates/grafana-stack/`) to
Kubernetes: same image versions, same configs, same pipeline behaviour. The
`ansible/` directory stays as the reference it was ported from.

Ledgerline itself still runs on `czy` with Docker Compose (see
`docs/roadmap/phase-1.md`). Only the platform lives here.

## What's deployed

Namespace `ledgerline-observability`. Service names match the compose service
names, so every in-config URL (`http://prometheus:9090/api/v1/write`,
`tempo:4317`, …) is unchanged from the Ansible templates.

| Component | Image | Endpoint | Auth |
|---|---|---|---|
| Grafana | `grafana/grafana:12.1.1` | https://grafana.vione.cloud | Grafana login |
| Prometheus | `prom/prometheus:v3.5.0` | https://prometheus.vione.cloud | basic auth |
| Alertmanager | `prom/alertmanager:v0.28.1` | https://alertmanager.vione.cloud | basic auth |
| Loki (query + push) | `grafana/loki:3.5.3` | https://loki.vione.cloud | basic auth |
| Tempo (query) | `grafana/tempo:2.8.2` | https://tempo.vione.cloud | basic auth |
| Alloy UI | `grafana/alloy:v1.10.2` | https://alloy.vione.cloud | basic auth |
| OTLP/HTTP ingest (Alloy :4318) | — | https://otlp.vione.cloud | basic auth |
| node-exporter (falken host) | `prom/node-exporter:v1.9.1` | cluster-internal | — |

OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.

### Differences from the Ansible stack, and why

- **Alloy's Docker-log half runs on czy, not here.** k3s uses containerd, so
  there's no Docker socket, and Ledgerline's containers are on czy anyway. The
  `discovery.docker` → `loki.source.docker` chain is in
  `observability/alloy-agent/config.alloy`, run by the `alloy-agent` service in
  `docker-compose.traefik.yml`. It pushes to `loki.vione.cloud`. The in-cluster
  Alloy keeps the OTLP receiver exactly as it was: **traces output only**.
- **Endpoints are subdomains behind Traefik + basic auth** instead of raw host
  ports. `vione.cloud` sits behind Cloudflare's proxy, which only forwards
  80/443 (plus a few others), so raw ports like `:3000` would never work.
- **Resource requests/limits** on every container (falken is shared).
- **App scrape job added:** `ledgerline` (the role's
  `prometheus_extra_scrape_jobs` is empty). Prometheus scrapes each czy
  service at `https://ledgerline.vione.cloud/metrics/<service>`, which czy's
  Traefik routes to that container's `:9464` behind basic auth. The scrape
  credentials are the `prometheus-scrape-auth` secret, mounted at
  `/etc/prometheus/secrets`.
- **Tempo labels exemplars `trace_id`** (`trace_id_label_name` in the
  overrides). Tempo's default is `traceID`, which the Prometheus datasource's
  `exemplarTraceIdDestinations` (`trace_id`) doesn't match, so span-metric
  exemplars rendered in Grafana but didn't link to the trace.

The CLAUDE.md platform facts still hold:
- OTLP metrics and logs are rejected: Alloy returns a 404 on `/v1/metrics` and
  `/v1/logs`, so the only sign is an export error in the app's own logs.
- Span metrics and the service graph come from Tempo's metrics-generator.
- The Loki derived field matches lowercase `trace_id`.
- Datasources are provisioned; dashboards are not.

## Deploy

```bash
# once: generate credentials (k8s/.credentials, gitignored) and create secrets.
# Also writes .env.observability and observability/traefik/metrics-users for czy.
./k8s/create-secrets.sh

# deploy, or re-apply after editing any config file
kubectl apply -k k8s/
```

Every config file goes through a `configMapGenerator`. Its name carries a
content hash, so editing e.g. `alloy/config.alloy` and re-applying rolls that
pod automatically. That's the equivalent of re-running the playbook.

If the API server isn't reachable from your machine, build locally and apply
on the node:

```bash
KUBECTL="ssh falken kubectl" ./k8s/create-secrets.sh
kubectl kustomize k8s/ | ssh falken kubectl apply -f -
```

**czy files** (once, then `./deploy.sh` keeps it running). Copy
`metrics-users` before the first deploy that mounts it. Otherwise Docker
creates a directory at that path, and Traefik's metrics routes fail until you
remove it:

```bash
scp .env.observability czy:/opt/apps/LedgerLine/.env.observability
ssh czy mkdir -p /opt/apps/LedgerLine/observability/traefik
scp observability/traefik/metrics-users czy:/opt/apps/LedgerLine/observability/traefik/metrics-users
./deploy.sh
```

## Teardown / rebuild

```bash
kubectl delete -k k8s/      # removes everything, including PVCs (all data)
```

`grafana-admin`, `obs-basic-auth` and `prometheus-scrape-auth` are deleted with the namespace. Run
`create-secrets.sh` again before re-applying; it reuses `k8s/.credentials`.

## Credentials

`k8s/.credentials` (mode 600, gitignored):
- `OBS_USER` / `OBS_PASSWORD`: basic auth for every endpoint except Grafana.
- `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD`: the Grafana login.

```bash
source k8s/.credentials
curl -s -u "$OBS_USER:$OBS_PASSWORD" https://prometheus.vione.cloud/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'
```

For an OpenTelemetry SDK (the header value must be URL-encoded, so the space
becomes `%20`):

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20$(printf '%s:%s' "$OBS_USER" "$OBS_PASSWORD" | base64)"
```

## Known gaps

- **czy host metrics:** node-exporter here watches falken. czy's own
  saturation (1 vCPU / 1 GB) is invisible until a node-exporter runs there
  and has a scrape path.