Port the observability platform to k3s and ship czy logs to it
k8s/ is now a faithful port of the mentor's Ansible Grafana stack (Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter) for the ledgerline-observability namespace on falken: same versions and configs, one *.vione.cloud ingress per tool behind Traefik basic auth, secrets generated by create-secrets.sh into gitignored files. Two fixes over the Ansible role, both verified live: Tempo's metrics-generator needs traces_storage or it runs no processors, and Prometheus needs exemplar-storage to keep exemplars. Ledgerline stays on czy; an Alloy agent in the Traefik override ships its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
34a11da848
commit
7719ef5fcf
+13
-7
@@ -62,10 +62,14 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
|
||||
and can be enabled per-tenant — at the cost of load on the trace store.
|
||||
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
|
||||
click a spike on a latency graph and land in the exact trace. On your platform
|
||||
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
|
||||
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
|
||||
`name: trace_id`. If you emit your own histograms with exemplars, the label must
|
||||
be `trace_id` to match.
|
||||
this is wired: Tempo's generator sets `send_exemplars: true`, Prometheus stores
|
||||
them (`--enable-feature=exemplar-storage`, added in `k8s/`; the Ansible role
|
||||
lacked it), and the Prometheus datasource declares `exemplarTraceIdDestinations`
|
||||
with `name: trace_id`. If you emit your own histograms with exemplars, the label
|
||||
must be `trace_id` to match. **But not quite end to end:** Tempo's generator
|
||||
labels its exemplars `traceID` (verified live), so the span-metric exemplars
|
||||
don't match that destination. Find the fix yourself and choose one: the
|
||||
datasource, Tempo, or both.
|
||||
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
|
||||
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
|
||||
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
|
||||
@@ -92,9 +96,11 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
|
||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
||||
metrics, so pick a path and implement it:
|
||||
- *Platform grain:* expose `/metrics` on each service and add them to
|
||||
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
|
||||
playbook. Prometheus scrapes your services directly. Note the irony and the
|
||||
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
|
||||
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
|
||||
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
|
||||
**First decide how Prometheus on falken reaches czy**: the Traefik
|
||||
override strips every service's host port (see `phase-1.md`). Note the irony and the
|
||||
lesson — you're back to a pull model and a second client library, which is
|
||||
exactly the fragmentation OTel exists to fix.
|
||||
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
|
||||
|
||||
Reference in New Issue
Block a user