Port the observability platform to k3s and ship czy logs to it
k8s/ is now a faithful port of the mentor's Ansible Grafana stack (Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter) for the ledgerline-observability namespace on falken: same versions and configs, one *.vione.cloud ingress per tool behind Traefik basic auth, secrets generated by create-secrets.sh into gitignored files. Two fixes over the Ansible role, both verified live: Tempo's metrics-generator needs traces_storage or it runs no processors, and Prometheus needs exemplar-storage to keep exemplars. Ledgerline stays on czy; an Alloy agent in the Traefik override ships its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
34a11da848
commit
7719ef5fcf
+117
@@ -0,0 +1,117 @@
|
||||
# Observability platform on k3s (`falken`)
|
||||
|
||||
The Grafana stack that Ledgerline reports to. It is a **port of the mentor's
|
||||
Ansible role** (`ansible/roles/observability/templates/grafana-stack/`) to
|
||||
Kubernetes: same image versions, same configs, same pipeline behaviour. The
|
||||
`ansible/` directory stays as the reference it was ported from.
|
||||
|
||||
Ledgerline itself still runs on `czy` with Docker Compose (see
|
||||
`docs/roadmap/phase-1.md`). Only the platform lives here.
|
||||
|
||||
## What's deployed
|
||||
|
||||
Namespace `ledgerline-observability`. Service names match the compose service
|
||||
names, so every in-config URL (`http://prometheus:9090/api/v1/write`,
|
||||
`tempo:4317`, …) is unchanged from the Ansible templates.
|
||||
|
||||
| Component | Image | Endpoint | Auth |
|
||||
|---|---|---|---|
|
||||
| Grafana | `grafana/grafana:12.1.1` | https://grafana.vione.cloud | Grafana login |
|
||||
| Prometheus | `prom/prometheus:v3.5.0` | https://prometheus.vione.cloud | basic auth |
|
||||
| Alertmanager | `prom/alertmanager:v0.28.1` | https://alertmanager.vione.cloud | basic auth |
|
||||
| Loki (query + push) | `grafana/loki:3.5.3` | https://loki.vione.cloud | basic auth |
|
||||
| Tempo (query) | `grafana/tempo:2.8.2` | https://tempo.vione.cloud | basic auth |
|
||||
| Alloy UI | `grafana/alloy:v1.10.2` | https://alloy.vione.cloud | basic auth |
|
||||
| OTLP/HTTP ingest (Alloy :4318) | — | https://otlp.vione.cloud | basic auth |
|
||||
| node-exporter (falken host) | `prom/node-exporter:v1.9.1` | cluster-internal | — |
|
||||
|
||||
OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.
|
||||
|
||||
### Differences from the Ansible stack, and why
|
||||
|
||||
- **Alloy's Docker-log half runs on czy, not here.** k3s uses containerd, so
|
||||
there's no Docker socket, and Ledgerline's containers are on czy anyway. The
|
||||
`discovery.docker` → `loki.source.docker` chain is in
|
||||
`observability/alloy-agent/config.alloy`, run by the `alloy-agent` service in
|
||||
`docker-compose.traefik.yml`. It pushes to `loki.vione.cloud`. The in-cluster
|
||||
Alloy keeps the OTLP receiver exactly as it was: **traces output only**.
|
||||
- **Endpoints are subdomains behind Traefik + basic auth** instead of raw host
|
||||
ports. `vione.cloud` sits behind Cloudflare's proxy, which only forwards
|
||||
80/443 (plus a few others), so raw ports like `:3000` would never work.
|
||||
- **Resource requests/limits** on every container (falken is shared).
|
||||
- **Not ported:** Prometheus app scrape jobs (`prometheus_extra_scrape_jobs` is
|
||||
empty in the role too; see `prometheus/prometheus.yml`).
|
||||
|
||||
The CLAUDE.md platform facts still hold:
|
||||
- OTLP metrics and logs are silently dropped.
|
||||
- Span metrics and the service graph come from Tempo's metrics-generator.
|
||||
- The Loki derived field matches lowercase `trace_id`.
|
||||
- Datasources are provisioned; dashboards are not.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
# once: generate credentials (k8s/.credentials, gitignored) and create secrets.
|
||||
# Also writes .env.observability for the czy agent.
|
||||
./k8s/create-secrets.sh
|
||||
|
||||
# deploy, or re-apply after editing any config file
|
||||
kubectl apply -k k8s/
|
||||
```
|
||||
|
||||
Every config file goes through a `configMapGenerator`. Its name carries a
|
||||
content hash, so editing e.g. `alloy/config.alloy` and re-applying rolls that
|
||||
pod automatically. That's the equivalent of re-running the playbook.
|
||||
|
||||
If the API server isn't reachable from your machine, build locally and apply
|
||||
on the node:
|
||||
|
||||
```bash
|
||||
KUBECTL="ssh falken kubectl" ./k8s/create-secrets.sh
|
||||
kubectl kustomize k8s/ | ssh falken kubectl apply -f -
|
||||
```
|
||||
|
||||
**czy agent** (once, then `./deploy.sh` keeps it running):
|
||||
|
||||
```bash
|
||||
scp .env.observability czy:/opt/apps/LedgerLine/.env.observability
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
## Teardown / rebuild
|
||||
|
||||
```bash
|
||||
kubectl delete -k k8s/ # removes everything, including PVCs (all data)
|
||||
```
|
||||
|
||||
`grafana-admin` and `obs-basic-auth` are deleted with the namespace. Run
|
||||
`create-secrets.sh` again before re-applying; it reuses `k8s/.credentials`.
|
||||
|
||||
## Credentials
|
||||
|
||||
`k8s/.credentials` (mode 600, gitignored):
|
||||
- `OBS_USER` / `OBS_PASSWORD`: basic auth for every endpoint except Grafana.
|
||||
- `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD`: the Grafana login.
|
||||
|
||||
```bash
|
||||
source k8s/.credentials
|
||||
curl -s -u "$OBS_USER:$OBS_PASSWORD" https://prometheus.vione.cloud/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'
|
||||
```
|
||||
|
||||
For an OpenTelemetry SDK (the header value must be URL-encoded, so the space
|
||||
becomes `%20`):
|
||||
|
||||
```bash
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20$(printf '%s:%s' "$OBS_USER" "$OBS_PASSWORD" | base64)"
|
||||
```
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **App metrics (Phase 5):** Prometheus here can't reach czy's services.
|
||||
Their host ports are stripped by the Traefik override. Decide the path
|
||||
before adding scrape jobs.
|
||||
- **czy host metrics:** node-exporter here watches falken. czy's own
|
||||
saturation (1 vCPU / 1 GB) is invisible until a node-exporter runs there
|
||||
and has a scrape path.
|
||||
@@ -0,0 +1,17 @@
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: default
|
||||
group_by: [alertname, instance]
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 4h
|
||||
|
||||
# No-op receiver (the Ansible role's default when alertmanager_webhook_url is
|
||||
# empty). To deliver alerts, add e.g.:
|
||||
# webhook_configs:
|
||||
# - url: "https://..."
|
||||
# send_resolved: true
|
||||
receivers:
|
||||
- name: default
|
||||
@@ -0,0 +1,46 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: alertmanager
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: alertmanager
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: alertmanager
|
||||
spec:
|
||||
containers:
|
||||
- name: alertmanager
|
||||
image: prom/alertmanager:v0.28.1
|
||||
args:
|
||||
- --config.file=/etc/alertmanager/alertmanager.yml
|
||||
- --storage.path=/alertmanager
|
||||
ports:
|
||||
- containerPort: 9093
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /-/ready
|
||||
port: 9093
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/alertmanager
|
||||
- name: data
|
||||
mountPath: /alertmanager
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: alertmanager-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: alertmanager-data
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: alertmanager
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
traefik.ingress.kubernetes.io/router.middlewares: ledgerline-observability-basic-auth@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- alertmanager.vione.cloud
|
||||
secretName: alertmanager-tls
|
||||
rules:
|
||||
- host: alertmanager.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: alertmanager
|
||||
port:
|
||||
number: 9093
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- pvc.yaml
|
||||
- service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: alertmanager-config
|
||||
files:
|
||||
- alertmanager.yml
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: alertmanager-data
|
||||
spec:
|
||||
storageClassName: local-path
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: alertmanager
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: alertmanager
|
||||
ports:
|
||||
- name: http
|
||||
port: 9093
|
||||
targetPort: 9093
|
||||
@@ -0,0 +1,35 @@
|
||||
// Port of ansible/roles/observability/templates/grafana-stack/config/alloy/config.alloy.j2.
|
||||
//
|
||||
// The Ansible config also tails container logs from the local Docker socket
|
||||
// (discovery.docker -> discovery.relabel -> loki.source.docker -> loki.write).
|
||||
// On k3s there is no Docker socket, and Ledgerline's containers run on czy
|
||||
// anyway, so that half lives in the czy-side agent:
|
||||
// observability/alloy-agent/config.alloy.
|
||||
//
|
||||
// What remains is unchanged: the OTLP receiver wires a `traces` output and
|
||||
// nothing else. OTLP metrics and logs are accepted and silently dropped.
|
||||
|
||||
logging {
|
||||
level = "info"
|
||||
}
|
||||
|
||||
otelcol.receiver.otlp "default" {
|
||||
grpc {
|
||||
endpoint = "0.0.0.0:4317"
|
||||
}
|
||||
http {
|
||||
endpoint = "0.0.0.0:4318"
|
||||
}
|
||||
output {
|
||||
traces = [otelcol.exporter.otlp.tempo.input]
|
||||
}
|
||||
}
|
||||
|
||||
otelcol.exporter.otlp "tempo" {
|
||||
client {
|
||||
endpoint = "tempo:4317"
|
||||
tls {
|
||||
insecure = true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: alloy
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: alloy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: alloy
|
||||
spec:
|
||||
containers:
|
||||
- name: alloy
|
||||
image: grafana/alloy:v1.10.2
|
||||
args:
|
||||
- run
|
||||
- --server.http.listen-addr=0.0.0.0:12345
|
||||
- --storage.path=/var/lib/alloy/data
|
||||
- /etc/alloy/config.alloy
|
||||
ports:
|
||||
- containerPort: 12345
|
||||
- containerPort: 4317
|
||||
- containerPort: 4318
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /-/ready
|
||||
port: 12345
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 384Mi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/alloy
|
||||
- name: data
|
||||
mountPath: /var/lib/alloy/data
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: alloy-config
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
@@ -0,0 +1,42 @@
|
||||
# Two hosts, both behind basic auth:
|
||||
# otlp.vione.cloud -> OTLP/HTTP ingest (:4318). Apps use
|
||||
# OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud,
|
||||
# OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf and an
|
||||
# Authorization header. OTLP/gRPC (:4317) is
|
||||
# cluster-internal only.
|
||||
# alloy.vione.cloud -> Alloy UI (:12345), the component graph.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: alloy
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
traefik.ingress.kubernetes.io/router.middlewares: ledgerline-observability-basic-auth@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- otlp.vione.cloud
|
||||
- alloy.vione.cloud
|
||||
secretName: alloy-tls
|
||||
rules:
|
||||
- host: otlp.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: alloy
|
||||
port:
|
||||
number: 4318
|
||||
- host: alloy.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: alloy
|
||||
port:
|
||||
number: 12345
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: alloy-config
|
||||
files:
|
||||
- config.alloy
|
||||
@@ -0,0 +1,18 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: alloy
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: alloy
|
||||
ports:
|
||||
- name: http
|
||||
port: 12345
|
||||
targetPort: 12345
|
||||
- name: otlp-grpc
|
||||
port: 4317
|
||||
targetPort: 4317
|
||||
- name: otlp-http
|
||||
port: 4318
|
||||
targetPort: 4318
|
||||
@@ -0,0 +1,5 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- middleware.yaml
|
||||
@@ -0,0 +1,14 @@
|
||||
# Shared basic auth for every ingress whose backend has no auth of its own
|
||||
# (Prometheus, Alertmanager, Loki, Tempo, Alloy UI, OTLP ingest). Grafana uses
|
||||
# its own login instead. The htpasswd secret is created by create-secrets.sh,
|
||||
# never committed.
|
||||
#
|
||||
# Ingresses reference it as:
|
||||
# traefik.ingress.kubernetes.io/router.middlewares: ledgerline-observability-basic-auth@kubernetescrd
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: Middleware
|
||||
metadata:
|
||||
name: basic-auth
|
||||
spec:
|
||||
basicAuth:
|
||||
secret: obs-basic-auth
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Create the platform's secrets in the cluster. Idempotent: credentials are
|
||||
# generated once into k8s/.credentials (gitignored, mode 600) and reused on
|
||||
# every later run, so re-running only re-applies the same secrets.
|
||||
#
|
||||
# ./k8s/create-secrets.sh
|
||||
#
|
||||
# Creates, in namespace ledgerline-observability:
|
||||
# obs-basic-auth htpasswd for the Traefik basic-auth middleware
|
||||
# (Prometheus, Alertmanager, Loki, Tempo, Alloy UI, OTLP)
|
||||
# grafana-admin Grafana admin user/password
|
||||
# and writes .env.observability at the repo root (gitignored) for the czy-side
|
||||
# Alloy log agent; copy it to the server once (see k8s/README.md).
|
||||
#
|
||||
# KUBECTL defaults to `kubectl`; e.g. KUBECTL="ssh falken kubectl" to run it on
|
||||
# the node when the API server isn't reachable from here.
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
KUBECTL="${KUBECTL:-kubectl}"
|
||||
NS=ledgerline-observability
|
||||
CREDS=.credentials
|
||||
|
||||
if [ ! -f "$CREDS" ]; then
|
||||
umask 077
|
||||
cat > "$CREDS" <<EOF
|
||||
OBS_USER=ledgerline
|
||||
OBS_PASSWORD=$(openssl rand -hex 24)
|
||||
GRAFANA_ADMIN_USER=admin
|
||||
GRAFANA_ADMIN_PASSWORD=$(openssl rand -hex 24)
|
||||
EOF
|
||||
echo "generated $PWD/$CREDS"
|
||||
fi
|
||||
# shellcheck disable=SC1090
|
||||
source "$CREDS"
|
||||
|
||||
# Secrets are rendered locally (no API access needed) and applied through
|
||||
# $KUBECTL, so they never appear on a command line on the remote side.
|
||||
$KUBECTL apply -f - < namespace.yaml
|
||||
|
||||
kubectl create secret generic obs-basic-auth -n "$NS" \
|
||||
--from-literal=users="$(htpasswd -nbB "$OBS_USER" "$OBS_PASSWORD")" \
|
||||
--dry-run=client -o yaml | $KUBECTL apply -f -
|
||||
|
||||
kubectl create secret generic grafana-admin -n "$NS" \
|
||||
--from-literal=admin-user="$GRAFANA_ADMIN_USER" \
|
||||
--from-literal=admin-password="$GRAFANA_ADMIN_PASSWORD" \
|
||||
--dry-run=client -o yaml | $KUBECTL apply -f -
|
||||
|
||||
(
|
||||
umask 077
|
||||
cat > ../.env.observability <<EOF
|
||||
# Credentials for the Alloy log agent on czy (docker-compose.traefik.yml).
|
||||
# Generated by k8s/create-secrets.sh -- do not commit.
|
||||
LOKI_PUSH_URL=https://loki.vione.cloud/loki/api/v1/push
|
||||
LOKI_PUSH_USER=$OBS_USER
|
||||
LOKI_PUSH_PASSWORD=$OBS_PASSWORD
|
||||
EOF
|
||||
)
|
||||
echo "wrote $(cd .. && pwd)/.env.observability"
|
||||
@@ -0,0 +1,53 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
editable: true
|
||||
jsonData:
|
||||
exemplarTraceIdDestinations:
|
||||
- datasourceUid: tempo
|
||||
name: trace_id
|
||||
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
editable: true
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
|
||||
name: TraceID
|
||||
url: '$${__value.raw}'
|
||||
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
editable: true
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
filterByTraceID: true
|
||||
spanStartTimeShift: -1m
|
||||
spanEndTimeShift: 1m
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
|
||||
- name: Alertmanager
|
||||
uid: alertmanager
|
||||
type: alertmanager
|
||||
access: proxy
|
||||
url: http://alertmanager:9093
|
||||
editable: true
|
||||
jsonData:
|
||||
implementation: prometheus
|
||||
@@ -0,0 +1,60 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: grafana
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: grafana
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
containers:
|
||||
- name: grafana
|
||||
image: grafana/grafana:12.1.1
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
env:
|
||||
- name: GF_SECURITY_ADMIN_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: grafana-admin
|
||||
key: admin-user
|
||||
- name: GF_SECURITY_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: grafana-admin
|
||||
key: admin-password
|
||||
- name: GF_USERS_ALLOW_SIGN_UP
|
||||
value: "false"
|
||||
- name: GF_FEATURE_TOGGLES_ENABLE
|
||||
value: traceqlEditor
|
||||
- name: GF_SERVER_ROOT_URL
|
||||
value: https://grafana.vione.cloud
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: 3000
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/grafana
|
||||
- name: datasources
|
||||
mountPath: /etc/grafana/provisioning/datasources
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: grafana-data
|
||||
- name: datasources
|
||||
configMap:
|
||||
name: grafana-datasources
|
||||
@@ -0,0 +1,24 @@
|
||||
# Grafana has its own login, so no basic-auth middleware here.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: grafana
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- grafana.vione.cloud
|
||||
secretName: grafana-tls
|
||||
rules:
|
||||
- host: grafana.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: grafana
|
||||
port:
|
||||
number: 3000
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- pvc.yaml
|
||||
- service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
# Datasources only, as in the Ansible role. Dashboard provisioning is the
|
||||
# Phase 6 deliverable and belongs in this directory.
|
||||
configMapGenerator:
|
||||
- name: grafana-datasources
|
||||
files:
|
||||
- datasources.yml
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: grafana-data
|
||||
spec:
|
||||
storageClassName: local-path
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: grafana
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: grafana
|
||||
ports:
|
||||
- name: http
|
||||
port: 3000
|
||||
targetPort: 3000
|
||||
@@ -0,0 +1,26 @@
|
||||
# Ledgerline observability platform on k3s (falken).
|
||||
#
|
||||
# A port of the mentor's Ansible Grafana stack
|
||||
# (ansible/roles/observability/templates/grafana-stack/): same versions, same
|
||||
# configs, same pipeline behaviour. Service names match the compose service
|
||||
# names so every in-config URL ports unchanged.
|
||||
#
|
||||
# ./create-secrets.sh # once: basic-auth + Grafana admin secrets
|
||||
# kubectl apply -k k8s/ # deploy / re-apply after a config change
|
||||
#
|
||||
# See README.md.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: ledgerline-observability
|
||||
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- auth
|
||||
- prometheus
|
||||
- alertmanager
|
||||
- loki
|
||||
- tempo
|
||||
- alloy
|
||||
- grafana
|
||||
- node-exporter
|
||||
@@ -0,0 +1,46 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: loki
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: loki
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: loki
|
||||
spec:
|
||||
containers:
|
||||
- name: loki
|
||||
image: grafana/loki:3.5.3
|
||||
args:
|
||||
- -config.file=/etc/loki/loki.yml
|
||||
ports:
|
||||
- containerPort: 3100
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3100
|
||||
initialDelaySeconds: 15
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 768Mi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/loki
|
||||
- name: data
|
||||
mountPath: /var/loki
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: loki-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: loki-data
|
||||
@@ -0,0 +1,26 @@
|
||||
# Query API and push endpoint (/loki/api/v1/push) on one host, both behind
|
||||
# basic auth. The Alloy log agent on czy pushes here.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: loki
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
traefik.ingress.kubernetes.io/router.middlewares: ledgerline-observability-basic-auth@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- loki.vione.cloud
|
||||
secretName: loki-tls
|
||||
rules:
|
||||
- host: loki.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: loki
|
||||
port:
|
||||
number: 3100
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- pvc.yaml
|
||||
- service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: loki-config
|
||||
files:
|
||||
- loki.yml
|
||||
@@ -0,0 +1,33 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
|
||||
common:
|
||||
path_prefix: /var/loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
instance_addr: 127.0.0.1
|
||||
kvstore:
|
||||
store: inmemory
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
storage_config:
|
||||
filesystem:
|
||||
directory: /var/loki/chunks
|
||||
|
||||
compactor:
|
||||
working_directory: /var/loki/compactor
|
||||
|
||||
limits_config:
|
||||
retention_period: 336h
|
||||
allow_structured_metadata: true
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: loki-data
|
||||
spec:
|
||||
storageClassName: local-path
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: loki
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: loki
|
||||
ports:
|
||||
- name: http
|
||||
port: 3100
|
||||
targetPort: 3100
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: ledgerline-observability
|
||||
@@ -0,0 +1,44 @@
|
||||
# Host metrics for falken (the observability host), as in the Ansible stack's
|
||||
# node-exporter (pid: host, / mounted at /host). hostNetwork so network stats
|
||||
# are the host's, not the pod's. Port 9100 on the host is blocked from the
|
||||
# internet by falken's ufw; Prometheus reaches it through the Service.
|
||||
#
|
||||
# This does NOT watch czy, where Ledgerline runs.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: node-exporter
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: node-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: node-exporter
|
||||
spec:
|
||||
hostPID: true
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
containers:
|
||||
- name: node-exporter
|
||||
image: prom/node-exporter:v1.9.1
|
||||
args:
|
||||
- --path.rootfs=/host
|
||||
ports:
|
||||
- containerPort: 9100
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 24Mi
|
||||
limits:
|
||||
memory: 96Mi
|
||||
volumeMounts:
|
||||
- name: host
|
||||
mountPath: /host
|
||||
readOnly: true
|
||||
mountPropagation: HostToContainer
|
||||
volumes:
|
||||
- name: host
|
||||
hostPath:
|
||||
path: /
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- daemonset.yaml
|
||||
- service.yaml
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: node-exporter
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: node-exporter
|
||||
ports:
|
||||
- name: http
|
||||
port: 9100
|
||||
targetPort: 9100
|
||||
@@ -0,0 +1,60 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: prometheus
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: prometheus
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: prometheus
|
||||
spec:
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: prom/prometheus:v3.5.0
|
||||
args:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
- --storage.tsdb.retention.time=15d
|
||||
- --web.enable-lifecycle
|
||||
- --web.enable-remote-write-receiver
|
||||
# Not in the Ansible template. Without it Prometheus discards the
|
||||
# exemplars Tempo's metrics-generator remote-writes
|
||||
# (send_exemplars: true), so the datasource's
|
||||
# exemplarTraceIdDestinations has nothing to link.
|
||||
- --enable-feature=exemplar-storage
|
||||
ports:
|
||||
- containerPort: 9090
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /-/ready
|
||||
port: 9090
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 1Gi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/prometheus/prometheus.yml
|
||||
subPath: prometheus.yml
|
||||
- name: rules
|
||||
mountPath: /etc/prometheus/rules
|
||||
- name: data
|
||||
mountPath: /prometheus
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: prometheus-config
|
||||
- name: rules
|
||||
configMap:
|
||||
name: prometheus-rules
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: prometheus-data
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: prometheus
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
traefik.ingress.kubernetes.io/router.middlewares: ledgerline-observability-basic-auth@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- prometheus.vione.cloud
|
||||
secretName: prometheus-tls
|
||||
rules:
|
||||
- host: prometheus.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: prometheus
|
||||
port:
|
||||
number: 9090
|
||||
@@ -0,0 +1,16 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- pvc.yaml
|
||||
- service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: prometheus-config
|
||||
files:
|
||||
- prometheus.yml
|
||||
- name: prometheus-rules
|
||||
files:
|
||||
- rules/host.rules.yml
|
||||
@@ -0,0 +1,41 @@
|
||||
global:
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
rule_files:
|
||||
- /etc/prometheus/rules/*.yml
|
||||
|
||||
alerting:
|
||||
alertmanagers:
|
||||
- static_configs:
|
||||
- targets: ["alertmanager:9093"]
|
||||
|
||||
scrape_configs:
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: ["prometheus:9090"]
|
||||
|
||||
- job_name: node-exporter
|
||||
static_configs:
|
||||
- targets: ["node-exporter:9100"]
|
||||
|
||||
- job_name: loki
|
||||
static_configs:
|
||||
- targets: ["loki:3100"]
|
||||
|
||||
- job_name: tempo
|
||||
static_configs:
|
||||
- targets: ["tempo:3200"]
|
||||
|
||||
- job_name: alloy
|
||||
static_configs:
|
||||
- targets: ["alloy:12345"]
|
||||
|
||||
- job_name: grafana
|
||||
static_configs:
|
||||
- targets: ["grafana:3000"]
|
||||
|
||||
# Application scrape jobs go here -- the equivalent of the Ansible role's
|
||||
# `prometheus_extra_scrape_jobs`. Ledgerline runs on czy with its service
|
||||
# ports unpublished, so a reachable path to /metrics has to be decided first
|
||||
# (Phase 5). Re-apply with `kubectl apply -k k8s/` after editing.
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: prometheus-data
|
||||
spec:
|
||||
storageClassName: local-path
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
@@ -0,0 +1,20 @@
|
||||
groups:
|
||||
- name: host.rules
|
||||
rules:
|
||||
- alert: HostExporterDown
|
||||
expr: up{job="node-exporter"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Node exporter is unavailable"
|
||||
description: "Prometheus has failed to scrape {{ $labels.instance }} for more than two minutes."
|
||||
|
||||
- alert: HostHighCPU
|
||||
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High CPU utilization"
|
||||
description: "CPU utilization on {{ $labels.instance }} has exceeded 85% for ten minutes."
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: prometheus
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: prometheus
|
||||
ports:
|
||||
- name: http
|
||||
port: 9090
|
||||
targetPort: 9090
|
||||
@@ -0,0 +1,48 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: tempo
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: tempo
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: tempo
|
||||
spec:
|
||||
containers:
|
||||
- name: tempo
|
||||
image: grafana/tempo:2.8.2
|
||||
args:
|
||||
- -config.file=/etc/tempo/tempo.yml
|
||||
ports:
|
||||
- containerPort: 3200
|
||||
- containerPort: 4317
|
||||
- containerPort: 4318
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3200
|
||||
initialDelaySeconds: 15
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 1Gi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/tempo
|
||||
- name: data
|
||||
mountPath: /var/tempo
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: tempo-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: tempo-data
|
||||
@@ -0,0 +1,26 @@
|
||||
# Tempo's query API (:3200) only. Apps send spans to Alloy (otlp.vione.cloud),
|
||||
# never to Tempo directly.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: tempo
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
traefik.ingress.kubernetes.io/router.middlewares: ledgerline-observability-basic-auth@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- tempo.vione.cloud
|
||||
secretName: tempo-tls
|
||||
rules:
|
||||
- host: tempo.vione.cloud
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: tempo
|
||||
port:
|
||||
number: 3200
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- deployment.yaml
|
||||
- pvc.yaml
|
||||
- service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: tempo-config
|
||||
files:
|
||||
- tempo.yml
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: tempo-data
|
||||
spec:
|
||||
storageClassName: local-path
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
@@ -0,0 +1,18 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: tempo
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: tempo
|
||||
ports:
|
||||
- name: http
|
||||
port: 3200
|
||||
targetPort: 3200
|
||||
- name: otlp-grpc
|
||||
port: 4317
|
||||
targetPort: 4317
|
||||
- name: otlp-http
|
||||
port: 4318
|
||||
targetPort: 4318
|
||||
@@ -0,0 +1,47 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
ingester:
|
||||
max_block_duration: 5m
|
||||
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 336h
|
||||
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
|
||||
metrics_generator:
|
||||
registry:
|
||||
external_labels:
|
||||
source: tempo
|
||||
storage:
|
||||
path: /var/tempo/generator/wal
|
||||
remote_write:
|
||||
- url: http://prometheus:9090/api/v1/write
|
||||
send_exemplars: true
|
||||
# Not in the Ansible template, which is broken without it: the
|
||||
# local-blocks processor needs a traces WAL, and when it can't initialise,
|
||||
# Tempo 2.8 starts *no* processors for the tenant ("local blocks processor
|
||||
# requires traces wal"), so service-graphs and span-metrics never emit.
|
||||
traces_storage:
|
||||
path: /var/tempo/generator/traces
|
||||
|
||||
overrides:
|
||||
defaults:
|
||||
metrics_generator:
|
||||
processors: [service-graphs, span-metrics, local-blocks]
|
||||
Reference in New Issue
Block a user