Port the observability platform to k3s and ship czy logs to it
k8s/ is now a faithful port of the mentor's Ansible Grafana stack (Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana, node-exporter) for the ledgerline-observability namespace on falken: same versions and configs, one *.vione.cloud ingress per tool behind Traefik basic auth, secrets generated by create-secrets.sh into gitignored files. Two fixes over the Ansible role, both verified live: Tempo's metrics-generator needs traces_storage or it runs no processors, and Prometheus needs exemplar-storage to keep exemplars. Ledgerline stays on czy; an Alloy agent in the Traefik override ships its container logs to Loki. Roadmap and CLAUDE.md now point at k8s/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
34a11da848
commit
7719ef5fcf
+49
-17
@@ -153,6 +153,20 @@ Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
|
||||
(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
|
||||
Elastic 9.1.3). Everything lands under `/opt/observability/`.
|
||||
|
||||
> **How it actually runs here (since 2026-10-12):** the Grafana stack is ported
|
||||
> to the k3s cluster `falken` as **`k8s/`** (same versions, configs and pipeline)
|
||||
> and deployed with `kubectl apply -k k8s/`; the Ansible project stays in
|
||||
> `ansible/` as the reference. Everything below describes the Ansible design and
|
||||
> still holds for the port. Where it names a template, the k8s equivalent is:
|
||||
> `config.alloy.j2` → `k8s/alloy/config.alloy` (plus the Docker-log half in
|
||||
> `observability/alloy-agent/config.alloy` on czy), `prometheus.yml.j2` /
|
||||
> `prometheus_extra_scrape_jobs` → `k8s/prometheus/prometheus.yml`,
|
||||
> `tempo.yml.j2` → `k8s/tempo/tempo.yml`, `datasources.yml.j2` →
|
||||
> `k8s/grafana/datasources.yml`, `alertmanager.yml.j2` →
|
||||
> `k8s/alertmanager/alertmanager.yml`, "re-run the playbook" →
|
||||
> `kubectl apply -k k8s/`. Deviations are listed in `k8s/README.md`. The ELK
|
||||
> stack is not ported.
|
||||
|
||||
### The architecture
|
||||
|
||||
Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
|
||||
@@ -266,6 +280,14 @@ names. The metric names differ from the OneUptime article's; go find the actual
|
||||
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
|
||||
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 9.)
|
||||
|
||||
*Verified live on 2026-10-12: as shipped, none of this worked.* `local-blocks`
|
||||
needs `metrics_generator.traces_storage`. Without it Tempo 2.8 logs "local blocks
|
||||
processor requires traces wal" and starts **no** processors, so there are no span
|
||||
metrics and no service graph. Separately, Prometheus throws away remote-written
|
||||
exemplars unless it runs with `--enable-feature=exemplar-storage`. Both are fixed
|
||||
in `k8s/`. It's worth telling Babak, and it's a good example of "config present
|
||||
≠ data arriving".
|
||||
|
||||
**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
|
||||
datasource is provisioned with:
|
||||
|
||||
@@ -285,10 +307,17 @@ datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram s
|
||||
this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
|
||||
in the Explore & Correlate column hangs off that one field name.
|
||||
|
||||
*Verified live:* Tempo's metrics-generator labels the exemplars it writes
|
||||
**`traceID`**, not `trace_id`. So the provisioned exemplar destination does not
|
||||
match the span-metric exemplars the platform itself produces, and clicking one
|
||||
won't open the trace. This is left unresolved on purpose (Phase 5 material).
|
||||
Work out the options and pick one.
|
||||
|
||||
**4. There is no dashboard provisioning at all.** The Grafana provisioning mount
|
||||
contains only `datasources/`. Every dashboard is yours to add — which means Babak's
|
||||
"create dashboards as code, destroy and recreate in seconds" requirement is entirely
|
||||
your deliverable, and the natural home for it is *inside this Ansible role*.
|
||||
your deliverable, and the natural home for it is *next to the platform's own
|
||||
provisioning*: originally the Ansible role, now `k8s/grafana/`.
|
||||
|
||||
### What this changes, phase by phase
|
||||
|
||||
@@ -296,38 +325,41 @@ your deliverable, and the natural home for it is *inside this Ansible role*.
|
||||
|---|---|
|
||||
| 0 / 0a | **None.** App and test suite are unaffected. |
|
||||
| 1 | **New.** Deploy Ledgerline itself to a server with Docker Compose behind Traefik. Plumbing only — no telemetry, no app changes. |
|
||||
| 2 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
|
||||
| 2 | Replaced. Deploy the real stack (ported to k3s as `k8s/`) instead of running a single container. |
|
||||
| 3 | Endpoint changes to Alloy; tracing work itself is identical. |
|
||||
| 4 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
|
||||
| 5 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
|
||||
| 6 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
|
||||
| 6 | **Upgraded.** Dashboards-as-code ships with the platform (`k8s/grafana/`, rebuilt by `kubectl apply -k k8s/`) — a better answer than a Makefile target. |
|
||||
| 7 | None. |
|
||||
| 8 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
|
||||
| 9 | None, plus TraceQL metrics from `local-blocks`. |
|
||||
| 10 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
|
||||
| 11 | Demo now includes `ansible-playbook` rebuilding the platform. |
|
||||
| 11 | Demo now includes `kubectl apply -k k8s/` rebuilding the platform. |
|
||||
|
||||
### Where to run it
|
||||
### Where it runs
|
||||
|
||||
The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
|
||||
ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside
|
||||
everything else will hurt. Use a separate VM (a cheap cloud instance or a local
|
||||
Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing
|
||||
at it. That also makes the deployment *real* rather than a localhost shortcut, which
|
||||
is the point of it being Ansible.
|
||||
ELK. Ledgerline runs on `czy` (1 vCPU / 1 GB, Phase 1), which has no room for
|
||||
either. The Grafana stack therefore runs on the single-node k3s cluster `falken`
|
||||
(4 CPU / 8 GB, shared with other workloads) in namespace
|
||||
`ledgerline-observability`, with resource requests and limits set. ELK alongside
|
||||
it would hurt; decide where it goes when Phase 10 comes.
|
||||
|
||||
If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker
|
||||
socket and will scrape **every** container's stdout, including Ledgerline's,
|
||||
automatically — convenient, and worth knowing before you wonder where those logs
|
||||
came from.
|
||||
In the Ansible design, Alloy's `loki.source.docker` reads the *platform host's*
|
||||
Docker socket. That only works when the apps run on that host. Here they don't,
|
||||
so that half of the config runs as a small Alloy agent on `czy`
|
||||
(`observability/alloy-agent/config.alloy`). It picks up **every** container's
|
||||
stdout there, including Traefik, Postgres and Redis, and pushes it to
|
||||
`https://loki.vione.cloud`.
|
||||
|
||||
### One thing to confirm with Babak
|
||||
|
||||
Is this repo a **platform you must consume as-is**, or a **starting point you may
|
||||
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
|
||||
assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
|
||||
extend `group_vars`. If Babak intends it as a fixed platform, several
|
||||
recommendations below change — ask before Phase 4.
|
||||
assumes you own it and will edit the Alloy config, add a dashboards provider, and
|
||||
add scrape jobs. If Babak intends it as a fixed platform, several
|
||||
recommendations below change — ask before Phase 4. Porting it to k3s (and the two
|
||||
config fixes) is already a modification worth mentioning to him.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -14,8 +14,8 @@ Babak gave you a task; deliver it like an engineer, not like a student.
|
||||
1. **`README.md`** — architecture diagram, one-command start, what each container is
|
||||
for, and a "here's the demo" script.
|
||||
2. **The five-minute demo path**, rehearsed:
|
||||
- Delete every dashboard in the Grafana UI, then `ansible-playbook
|
||||
playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
|
||||
- Delete every dashboard in the Grafana UI (or the whole namespace), then
|
||||
`kubectl apply -k k8s/` — the whole platform and every dashboard rebuilt
|
||||
from code, one command.
|
||||
- Submit one transaction from the frontend, then follow it through Grafana: RED
|
||||
dashboard → latency spike → exemplar → trace → span → logs for that span → back
|
||||
|
||||
+52
-32
@@ -57,38 +57,55 @@ than abstract.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||||
`ansible-galaxy collection install -r requirements.yml`, change
|
||||
`grafana_admin_password` (vault it), then
|
||||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||||
1. **Deploy the platform.** *Deployed 2026-10-12.* The mentor's Ansible
|
||||
Grafana stack was ported to the k3s cluster `falken` as `k8s/`, keeping the
|
||||
same versions, configs and pipeline. Deploy with `kubectl apply -k k8s/`.
|
||||
`k8s/README.md` covers deploy/teardown, credentials and every deviation
|
||||
from the Ansible role.
|
||||
|
||||
**Where the platform lives — not on the Ledgerline server.** Grafana (3000)
|
||||
and the rest come from the Ansible stack, not from Ledgerline's compose
|
||||
files, so `ledgerline.vione.cloud:3000` will never show Grafana. Three
|
||||
reasons it belongs on its own host:
|
||||
- `czy` has 1 GB RAM and Ledgerline already fills it; the platform README
|
||||
recommends 2 vCPU / 4 GB for the Grafana stack.
|
||||
- Phase 1's rule is that only 22/80/443 are open on `czy`; the platform
|
||||
publishes seven more ports.
|
||||
- `ledgerline.vione.cloud` is behind Cloudflare's proxy, which only forwards
|
||||
ports 80, 8080, 8880, 2052, 2082, 2086, 2095 (HTTP) and 443, 2053, 2083,
|
||||
2087, 2096, 8443 (HTTPS). Port 3000 (and 3100, 3200, 4318, 9090, 12345)
|
||||
never reaches the origin through a proxied hostname.
|
||||
**Why not the Ansible route on a VM:**
|
||||
- `czy` (1 GB) can't host it. The platform README recommends 2 vCPU / 4 GB.
|
||||
- `*.vione.cloud` sits behind Cloudflare's proxy, which only forwards a few
|
||||
ports (80/443, 8080, 8443, …), so raw ports like `:3000` never reach any
|
||||
origin. That's why `ledgerline.vione.cloud:3000` showed nothing.
|
||||
|
||||
So reach the platform by the obs host's IP, an SSH tunnel, or a separate
|
||||
**DNS-only** (grey-cloud) record. The platform's ports are published on all
|
||||
interfaces with little or no auth — firewall them to your own IP plus
|
||||
`czy`'s IP (which needs 4318 for OTLP). `<obs-host>` below means that host,
|
||||
reached from `czy` over the internet.
|
||||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||||
a single container with everything pre-wired. Useful for a first trace, but
|
||||
don't build on it — its pipeline differs from your real platform in exactly the
|
||||
ways that matter.)*
|
||||
**Endpoints:** each one is a subdomain behind Traefik. Everything except
|
||||
Grafana uses basic auth (`k8s/.credentials`).
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Grafana | https://grafana.vione.cloud |
|
||||
| Prometheus | https://prometheus.vione.cloud |
|
||||
| Alertmanager | https://alertmanager.vione.cloud |
|
||||
| Loki | https://loki.vione.cloud |
|
||||
| Tempo (query) | https://tempo.vione.cloud |
|
||||
| Alloy UI | https://alloy.vione.cloud |
|
||||
| OTLP/HTTP ingest | https://otlp.vione.cloud |
|
||||
|
||||
Container logs from `czy` reach Loki through an Alloy agent there (the
|
||||
`alloy-agent` service in `docker-compose.traefik.yml`).
|
||||
|
||||
**Verified live:** all of the following worked, using a scratch trace sent
|
||||
from `czy`.
|
||||
- Every endpoint answers, and returns 401 without auth.
|
||||
- Prometheus scrapes all six platform targets.
|
||||
- Traces flow czy → Alloy → Tempo.
|
||||
- Tempo's metrics-generator emits `traces_spanmetrics_*` and
|
||||
`traces_service_graph_*` with exemplars.
|
||||
- czy container logs land in Loki.
|
||||
|
||||
**Two platform bugs, fixed in the port.** As shipped, the Ansible role
|
||||
produced no span metrics or service graph (Tempo needs
|
||||
`metrics_generator.traces_storage`), and Prometheus discarded exemplars
|
||||
(missing `--enable-feature=exemplar-storage`).
|
||||
|
||||
**One issue left open on purpose:** Tempo labels its exemplars `traceID`,
|
||||
but the datasource's `exemplarTraceIdDestinations` expects `trace_id`.
|
||||
|
||||
Now open the Alloy UI and **read the component graph**. It shows you the
|
||||
exact pipeline, including the fact that the OTLP receiver has only a traces
|
||||
output. Seeing that in the UI before you get bitten by it is worth ten
|
||||
minutes.
|
||||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||||
auto-instrumentation:
|
||||
```
|
||||
@@ -98,9 +115,12 @@ than abstract.
|
||||
and change its command to
|
||||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||||
with `OTEL_SERVICE_NAME=sanctions`,
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vione.cloud`,
|
||||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||||
Tempo directly.
|
||||
Tempo directly — plus the basic-auth header
|
||||
`OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 user:pass>`
|
||||
(URL-encoded: the space is `%20`; see `k8s/README.md`). Keep the
|
||||
credential in the gitignored `.env.observability` on czy, not in compose.
|
||||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||||
|
||||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||||
|
||||
@@ -72,17 +72,20 @@ means anything).
|
||||
3. Add a redaction processor with a denylist of field names.
|
||||
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
|
||||
with a `container` label (Alloy's relabel rule sets it from the Docker container
|
||||
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
|
||||
name, e.g. `ledgerline-gateway-1`). Here that's the Alloy agent on czy
|
||||
(`observability/alloy-agent/config.alloy`, already running), not the
|
||||
cluster's Alloy. Verify the `trace_id` field is present *and lowercase* in
|
||||
the JSON body.
|
||||
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
|
||||
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
|
||||
the receiver shows traffic, and the pipeline shows no logs output. Now extend
|
||||
`config.alloy.j2` — add a `logs` output on the receiver plus an
|
||||
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
|
||||
`k8s/alloy/config.alloy` — add a `logs` output on the receiver plus an
|
||||
`otelcol.exporter.loki` → `loki.write` chain — `kubectl apply -k k8s/`, and watch
|
||||
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
|
||||
That is the most transferable half-hour in the whole project.
|
||||
6. Confirm the provisioned correlation works in both directions: Loki's
|
||||
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
|
||||
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
|
||||
(`filterByTraceID: true`) → Loki. Both are already in `k8s/grafana/datasources.yml`; you
|
||||
are verifying, not writing. If a link doesn't appear, the regex didn't match
|
||||
your field name — check that before anything else.
|
||||
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
|
||||
|
||||
+13
-7
@@ -62,10 +62,14 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
|
||||
and can be enabled per-tenant — at the cost of load on the trace store.
|
||||
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
|
||||
click a spike on a latency graph and land in the exact trace. On your platform
|
||||
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
|
||||
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
|
||||
`name: trace_id`. If you emit your own histograms with exemplars, the label must
|
||||
be `trace_id` to match.
|
||||
this is wired: Tempo's generator sets `send_exemplars: true`, Prometheus stores
|
||||
them (`--enable-feature=exemplar-storage`, added in `k8s/`; the Ansible role
|
||||
lacked it), and the Prometheus datasource declares `exemplarTraceIdDestinations`
|
||||
with `name: trace_id`. If you emit your own histograms with exemplars, the label
|
||||
must be `trace_id` to match. **But not quite end to end:** Tempo's generator
|
||||
labels its exemplars `traceID` (verified live), so the span-metric exemplars
|
||||
don't match that destination. Find the fix yourself and choose one: the
|
||||
datasource, Tempo, or both.
|
||||
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
|
||||
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
|
||||
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
|
||||
@@ -92,9 +96,11 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
|
||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
||||
metrics, so pick a path and implement it:
|
||||
- *Platform grain:* expose `/metrics` on each service and add them to
|
||||
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
|
||||
playbook. Prometheus scrapes your services directly. Note the irony and the
|
||||
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
|
||||
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
|
||||
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
|
||||
**First decide how Prometheus on falken reaches czy**: the Traefik
|
||||
override strips every service's host port (see `phase-1.md`). Note the irony and the
|
||||
lesson — you're back to a pull model and a second client library, which is
|
||||
exactly the fragmentation OTel exists to fix.
|
||||
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
|
||||
|
||||
+17
-12
@@ -45,22 +45,27 @@ recreate in seconds.*
|
||||
|
||||
Your platform provisions **datasources only** — there is no dashboards provider at
|
||||
all. So this phase is entirely yours, and the right home for it is inside the
|
||||
Ansible role, using the same mechanism as everything else.
|
||||
platform's own manifests (`k8s/grafana/`, the k3s port of the Ansible role),
|
||||
using the same mechanism as everything else.
|
||||
|
||||
1. **Extend the role.** Add
|
||||
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
|
||||
(a file-based dashboard provider pointing at a mounted directory), a
|
||||
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
|
||||
and copy them. The Compose template already mounts
|
||||
`./config/grafana/provisioning` read-only, so the provider is picked up on
|
||||
restart.
|
||||
1. **Extend `k8s/grafana/`.** Add a file-based dashboard provider (a
|
||||
`provisioning/dashboards/` YAML pointing at a mounted directory) and the
|
||||
dashboard JSON. Ship both through `configMapGenerator`, exactly as
|
||||
`datasources.yml` is shipped today, and mount them in `deployment.yaml`.
|
||||
Two k8s-specific things to work out:
|
||||
- A ConfigMap is capped at about 1 MiB, and client-side `kubectl apply`
|
||||
also stores a copy in an annotation. Large generated dashboards push
|
||||
against that.
|
||||
- The configMap hash suffix rolls the Grafana pod on every change.
|
||||
Decide whether that's what you want.
|
||||
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
|
||||
Foundation SDK (Python)**: one function producing a RED dashboard for a given
|
||||
service name, looped over the seven services; a second producing a system
|
||||
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
|
||||
the generator *and* its output; the Ansible role ships the output.
|
||||
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
|
||||
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
|
||||
the generator *and* its output; `k8s/grafana/` ships the output.
|
||||
3. **The demo is now `kubectl apply -k k8s/`.** Delete every dashboard through
|
||||
the Grafana UI, re-apply (or go further: `kubectl delete -k k8s/` and rebuild
|
||||
the whole platform), and watch them all come back.
|
||||
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
|
||||
than a local Makefile target, because it's the same command that builds the
|
||||
platform.
|
||||
@@ -71,6 +76,6 @@ Ansible role, using the same mechanism as everything else.
|
||||
wouldn't be.
|
||||
5. Add alert rules — see Phase 8 for the Alertmanager-vs-Grafana-alerting choice.
|
||||
|
||||
**Done when:** re-running the playbook restores every dashboard from scratch, and no
|
||||
**Done when:** `kubectl apply -k k8s/` restores every dashboard from scratch, and no
|
||||
dashboard has ever been saved through the Grafana UI.
|
||||
|
||||
|
||||
@@ -73,8 +73,10 @@ come from measured numbers, not be picked from thin air).
|
||||
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
|
||||
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
|
||||
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
|
||||
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
|
||||
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
|
||||
*Prometheus rule files → Alertmanager*. In the k3s port, rule files live in
|
||||
`k8s/prometheus/rules/` (add each one to the `prometheus-rules` generator)
|
||||
and delivery is a receiver in `k8s/alertmanager/alertmanager.yml` (the role's
|
||||
`alertmanager_webhook_url`). Follow the grain — but
|
||||
be able to argue the alternative: Grafana unified alerting can evaluate across
|
||||
datasources (including Loki queries) and lives with your dashboards, whereas
|
||||
Alertmanager is metrics-only but is the ecosystem standard and gives you
|
||||
|
||||
@@ -21,7 +21,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
||||
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
||||
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
|
||||
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
|
||||
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
|
||||
| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
|
||||
| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
|
||||
| 9 | [`phase-9.md`](phase-9.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
|
||||
@@ -37,8 +37,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
Traefik and a Let's Encrypt certificate, served at a real domain. Plumbing
|
||||
only — no telemetry, no app changes. **Done** — see `phase-1.md`.
|
||||
- **2 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
|
||||
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
|
||||
Ansible and get one auto-instrumented trace into Tempo.
|
||||
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform
|
||||
(the mentor's Ansible stack, ported to k3s as `k8s/` — already deployed on
|
||||
`falken`) and get one auto-instrumented trace into Tempo.
|
||||
- **3 — Distributed tracing.** The heart of the task: auto- and
|
||||
manual-instrument all seven services, and hand-propagate trace context
|
||||
across the Redis Stream hop, which carries none by default.
|
||||
@@ -50,8 +51,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
silently. Tempo's metrics-generator already produces span metrics and the
|
||||
service graph — don't duplicate it in the collector.
|
||||
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
||||
Grafana Foundation SDK and ship it through the Ansible role, so "destroy
|
||||
and recreate in seconds" is a real, repeatable command.
|
||||
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
|
||||
"destroy and recreate in seconds" is a real, repeatable command.
|
||||
- **7 — Load testing with k6, and percentiles.** Learn why percentiles can't
|
||||
be averaged, what coordinated omission is, and drive Ledgerline with
|
||||
arrival-rate executors to get a clean before/after load profile.
|
||||
|
||||
Reference in New Issue
Block a user