mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Port the observability platform to k3s and ship czy logs to it

Browse Source 
k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 13:24:38 +03:30
co-authored by Claude Opus 5.5
parent 34a11da848
commit 7719ef5fcf
57 changed files with 1507 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/roadmap/intro.md
+49 -17
View File
@@ -153,6 +153,20 @@ Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
Elastic 9.1.3). Everything lands under `/opt/observability/`.
> **How it actually runs here (since 2026-10-12):** the Grafana stack is ported
> to the k3s cluster `falken` as **`k8s/`** (same versions, configs and pipeline)
> and deployed with `kubectl apply -k k8s/`; the Ansible project stays in
> `ansible/` as the reference. Everything below describes the Ansible design and
> still holds for the port. Where it names a template, the k8s equivalent is:
> `config.alloy.j2` → `k8s/alloy/config.alloy` (plus the Docker-log half in
> `observability/alloy-agent/config.alloy` on czy), `prometheus.yml.j2` /
> `prometheus_extra_scrape_jobs` → `k8s/prometheus/prometheus.yml`,
> `tempo.yml.j2` → `k8s/tempo/tempo.yml`, `datasources.yml.j2` →
> `k8s/grafana/datasources.yml`, `alertmanager.yml.j2` →
> `k8s/alertmanager/alertmanager.yml`, "re-run the playbook" →
> `kubectl apply -k k8s/`. Deviations are listed in `k8s/README.md`. The ELK
> stack is not ported.
### The architecture
Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
@@ -266,6 +280,14 @@ names. The metric names differ from the OneUptime article's; go find the actual
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.)
*Verified live on 2026-10-12: as shipped, none of this worked.* `local-blocks`
needs `metrics_generator.traces_storage`. Without it Tempo 2.8 logs "local blocks
processor requires traces wal" and starts **no** processors, so there are no span
metrics and no service graph. Separately, Prometheus throws away remote-written
exemplars unless it runs with `--enable-feature=exemplar-storage`. Both are fixed
in `k8s/`. It's worth telling Babak, and it's a good example of "config present
≠ data arriving".
**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
datasource is provisioned with:
@@ -285,10 +307,17 @@ datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram s
this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
in the Explore & Correlate column hangs off that one field name.
*Verified live:* Tempo's metrics-generator labels the exemplars it writes
**`traceID`**, not `trace_id`. So the provisioned exemplar destination does not
match the span-metric exemplars the platform itself produces, and clicking one
won't open the trace. This is left unresolved on purpose (Phase 5 material).
Work out the options and pick one.
**4. There is no dashboard provisioning at all.** The Grafana provisioning mount
contains only `datasources/`. Every dashboard is yours to add — which means Babak's
"create dashboards as code, destroy and recreate in seconds" requirement is entirely
your deliverable, and the natural home for it is *inside this Ansible role*.
your deliverable, and the natural home for it is *next to the platform's own
provisioning*: originally the Ansible role, now `k8s/grafana/`.
### What this changes, phase by phase
@@ -296,38 +325,41 @@ your deliverable, and the natural home for it is *inside this Ansible role*.
|---|---|
| 0 / 0a | **None.** App and test suite are unaffected. |
| 1 | **New.** Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes. |
| 2 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
| 2 | Replaced. Deploy the real stack (ported to k3s as `k8s/`) instead of running a single container. |
| 3 | Endpoint changes to Alloy; tracing work itself is identical. |
| 4 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
| 5 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
| 6 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
| 6 | **Upgraded.** Dashboards-as-code ships with the platform (`k8s/grafana/`, rebuilt by `kubectl apply -k k8s/`) — a better answer than a Makefile target. |
| 7 | None. |
| 8 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
| 9 | None, plus TraceQL metrics from `local-blocks`. |
| 10 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
| 11 | Demo now includes `ansible-playbook` rebuilding the platform. |
| 11 | Demo now includes `kubectl apply -k k8s/` rebuilding the platform. |
### Where to run it
### Where it runs
The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside
everything else will hurt. Use a separate VM (a cheap cloud instance or a local
Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing
at it. That also makes the deployment *real* rather than a localhost shortcut, which
is the point of it being Ansible.
ELK. Ledgerline runs on `czy` (1 vCPU / 1 GB, Phase 1), which has no room for
either. The Grafana stack therefore runs on the single-node k3s cluster `falken`
(4 CPU / 8 GB, shared with other workloads) in namespace
`ledgerline-observability`, with resource requests and limits set. ELK alongside
it would hurt; decide where it goes when Phase 10 comes.
If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker
socket and will scrape **every** container's stdout, including Ledgerline's,
automatically — convenient, and worth knowing before you wonder where those logs
came from.
In the Ansible design, Alloy's `loki.source.docker` reads the *platform host's*
Docker socket. That only works when the apps run on that host. Here they don't,
so that half of the config runs as a small Alloy agent on `czy`
(`observability/alloy-agent/config.alloy`). It picks up **every** container's
stdout there, including Traefik, Postgres and Redis, and pushes it to
`https://loki.vione.cloud`.
### One thing to confirm with Babak
Is this repo a **platform you must consume as-is**, or a **starting point you may
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
extend `group_vars`. If Babak intends it as a fixed platform, several
recommendations below change — ask before Phase 4.
assumes you own it and will edit the Alloy config, add a dashboards provider, and
add scrape jobs. If Babak intends it as a fixed platform, several
recommendations below change — ask before Phase 4. Porting it to k3s (and the two
config fixes) is already a modification worth mentioning to him.
---