Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.
Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
czy publishes no service ports, so Traefik exposes each service's :9464
as https://ledgerline.vione.cloud/metrics/<service> behind basic auth,
and the platform's Prometheus gets a `ledgerline` job with one target
per service. The targets share an address; a `service` label tells them
apart, and relabels derive __metrics_path__ and instance from it.
create-secrets.sh now also creates the prometheus-scrape-auth secret
(mounted optional, so a missing secret fails only this job) and writes
the gitignored htpasswd file Traefik reads, which is copied to czy like
.env.observability.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.
Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.
Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Movses AslanyanandClaude Sonnet 5.5