mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Port the observability platform to k3s and ship czy logs to it

Browse Source 
k8s/ is now a faithful port of the mentor's Ansible Grafana stack
(Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter)
for the ledgerline-observability namespace on falken: same versions and
configs, one *.vione.cloud ingress per tool behind Traefik basic auth,
secrets generated by create-secrets.sh into gitignored files.

Two fixes over the Ansible role, both verified live: Tempo's
metrics-generator needs traces_storage or it runs no processors, and
Prometheus needs exemplar-storage to keep exemplars.

Ledgerline stays on czy; an Alloy agent in the Traefik override ships
its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 13:24:38 +03:30
co-authored by Claude Opus 5.5
parent 34a11da848
commit 7719ef5fcf
57 changed files with 1507 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+61 -29
View File
@@ -25,16 +25,30 @@ below. Phase 0 is built test-first from
two specs: `docs/phase-0a-test-suite-spec.md` (the tests) and
`docs/phase-0-base-project-spec.md` (the system).
### The observability backend is external and pre-built
### The observability backend is pre-built, ported to k3s
The Grafana stack is **not** built in this repo. It is deployed by a separate
Ansible project, `observability-ansible`, which the user received from his mentor.
It deploys two stacks on a target host via Docker Compose:
The platform came from the mentor as an Ansible project, `observability-ansible`
(kept in `ansible/` as the reference). It deploys two stacks on one host via
Docker Compose:
- **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with
**Grafana Alloy** as the collector and node-exporter for host metrics.
- **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the
hands-on subject of Phase 10, not part of the main pipeline.
hands-on subject of Phase 10, not part of the main pipeline. **Not ported.**
**The Grafana stack actually runs on the user's k3s cluster (`falken`), not via
Ansible.** `k8s/` is a faithful port of the role: same versions, configs and
pipeline, namespace `ledgerline-observability`, deployed with
`kubectl apply -k k8s/` (`k` is the user's alias with the right kubeconfig; if
the API server is unreachable, `kubectl kustomize k8s/ | ssh falken kubectl
apply -f -`). Endpoints are `https://{grafana,prometheus,alertmanager,loki,tempo,alloy,otlp}.vione.cloud`;
all but Grafana sit behind basic auth with credentials in `k8s/.credentials`.
Ledgerline stays on `czy` (Compose); its container logs reach Loki via an Alloy
agent there (`observability/alloy-agent/config.alloy`, in
`docker-compose.traefik.yml`) — the Docker-socket half of the original Alloy
config. `k8s/README.md` lists every deviation from the Ansible role. Where a
roadmap file still says "Ansible role" or "re-run the playbook", read it as
"`k8s/`" and "`kubectl apply -k k8s/`".
Five facts about that platform that change what correct code looks like here. Verify
them against the live system rather than assuming — they were read from the configs
@@ -47,8 +61,10 @@ at one point in time and the user may have since extended them.
separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
metrics "should be arriving"; check Prometheus or the Alloy UI first.
2. **App metrics reach Prometheus by direct scrape, not through Alloy.** Alloy has
no `prometheus.scrape` component; `prometheus.yml.j2` scrapes the stack's own
components plus whatever is listed in `prometheus_extra_scrape_jobs`. Note that
no `prometheus.scrape` component; `k8s/prometheus/prometheus.yml` scrapes the
stack's own components, and app jobs get added there (the role's
`prometheus_extra_scrape_jobs`). czy's service ports are unpublished, so no app
scrape path exists yet — a Phase 5 decision. Note that
the diagram draws this differently (app metrics routed via Alloy) — the diagram
and the code have drifted, and the user knows. Don't "correct" one to the other
unasked.
@@ -57,15 +73,20 @@ at one point in time and the user may have since extended them.
Prometheus with `send_exemplars: true`) — **not** from collector connectors.
Never add `spanmetrics` / `servicegraph` to Alloy; it duplicates every series.
The metric names differ from the ones in most blog posts — look them up in
Prometheus before writing PromQL.
Prometheus before writing PromQL. (As shipped, the Ansible role produced none of
this: Tempo lacked `traces_storage` and Prometheus lacked
`--enable-feature=exemplar-storage`. Both are fixed in `k8s/`; verified live.)
4. **Log→trace correlation runs on a case-sensitive regex:**
`trace[_-]?id["=: ]+([a-fA-F0-9]+)`. Logs must carry **lowercase `trace_id`**.
OpenTelemetry's logging auto-instrumentation emits `otelTraceID`, which does not
match. Exemplar labels must also be `trace_id`. The diagram names this outright:
"Correlation key: trace_id / exemplars."
"Correlation key: trace_id / exemplars." **Known, unresolved:** Tempo's
metrics-generator labels its exemplars `traceID` (verified live), which the
provisioned `exemplarTraceIdDestinations: name: trace_id` does not match — so
span-metric exemplars don't link to Tempo yet. The user decides the fix.
5. **There is no dashboard provisioning in the platform** — datasources only.
Dashboards-as-code is the user's deliverable and belongs in the Ansible role, so
that `ansible-playbook` rebuilds them.
Dashboards-as-code is the user's deliverable and belongs in `k8s/grafana/`, so
that `kubectl apply -k k8s/` rebuilds them.
Ledgerline talks to this platform over the network. Endpoints come from env
(`OTEL_EXPORTER_OTLP_ENDPOINT` etc.), never hardcoded.
@@ -294,42 +315,53 @@ git diff --stat tests/ # did anything touch the contract? should
The pathology suite (`make test-pathology`, `make test-slow`) is slow and is an
answer key — see spoiler rule 3 before running it.
**Traces (Tempo, :3200)** — search by service and duration rather than trusting
**The platform lives on falken** (see the backend section above). Load the
credentials first; every endpoint except Grafana needs basic auth:
```bash
source k8s/.credentials; A="$OBS_USER:$OBS_PASSWORD"
k get pods -n ledgerline-observability # what's actually running
```
**Traces (Tempo)** — search by service and duration rather than trusting
that a span exists:
```bash
curl -s 'http://localhost:3200/api/search?tags=service.name%3Dledger&limit=5'
curl -s "http://localhost:3200/api/traces/<trace_id>" | jq '.batches | length'
curl -s -u "$A" 'https://tempo.vione.cloud/api/search?tags=service.name%3Dledger&limit=5'
curl -s -u "$A" "https://tempo.vione.cloud/api/traces/<trace_id>" | jq '.batches | length'
```
**Metrics (Prometheus, :9090)** — and this is the cardinality check worth
**Metrics (Prometheus)** — and this is the cardinality check worth
running often:
```bash
curl -s 'http://localhost:9090/api/v1/query?query=count({__name__=~"ledgerline.*"})'
curl -s 'http://localhost:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'
curl -s -u "$A" 'https://prometheus.vione.cloud/api/v1/query?query=count({__name__=~"ledgerline.*"})'
curl -s -u "$A" 'https://prometheus.vione.cloud/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'
```
**Logs (Loki, :3100)** — confirm `trace_id` is actually in the payload, not just
in the code that was supposed to put it there:
**Logs (Loki)** — confirm `trace_id` is actually in the payload, not just
in the code that was supposed to put it there. The czy agent labels streams with
the full container name (`container` and `service_name` are both e.g.
`ledgerline-gateway-1`):
```bash
curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
--data-urlencode 'query={service_name="gateway"} | json | trace_id != ""' \
curl -s -u "$A" -G 'https://loki.vione.cloud/loki/api/v1/query_range' \
--data-urlencode 'query={container="ledgerline-gateway-1"} | json | trace_id != ""' \
--data-urlencode 'limit=5'
```
**Alloy (:12345)** — the component graph is the fastest way to see what the pipeline
actually does, including which signal outputs are wired. Check it before theorising
about missing data.
**Alloy (https://alloy.vione.cloud)** — the component graph is the fastest way to
see what the pipeline actually does, including which signal outputs are wired.
Check it before theorising about missing data. The czy log agent has no UI
exposed; `ssh czy docker logs ledgerline-alloy-agent-1`.
**Metric names** — never assume; the Tempo-generated names differ from blog posts:
```bash
curl -s 'http://<obs-host>:9090/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))'
curl -s -u "$A" 'https://prometheus.vione.cloud/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))'
```
**Grafana (:3000)** — verify provisioning actually loaded rather than reading
the YAML and assuming:
**Grafana (https://grafana.vione.cloud)** — verify provisioning actually loaded
rather than reading the YAML and assuming:
```bash
curl -s -u admin:admin 'http://localhost:3000/api/datasources' | jq '.[].name'
curl -s -u admin:admin 'http://localhost:3000/api/search?type=dash-db' | jq '.[].title'
G="$GRAFANA_ADMIN_USER:$GRAFANA_ADMIN_PASSWORD"
curl -s -u "$G" 'https://grafana.vione.cloud/api/datasources' | jq '.[].name'
curl -s -u "$G" 'https://grafana.vione.cloud/api/search?type=dash-db' | jq '.[].title'
```
Rules of thumb: