Add deployment phase and Traefik compose override
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b2112e55e1
commit
5e514ecf25
+109
-59
@@ -1,76 +1,126 @@
|
||||
# Phase 5 — Dashboards and alerts as code
|
||||
# Phase 5 — Metrics
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
|
||||
dashboarding.
|
||||
**Message:** Not blocked technically (metrics don't strictly need tracing
|
||||
first), but sequenced after Phase 4 per the roadmap's ordering.
|
||||
|
||||
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
|
||||
recreate in seconds.*
|
||||
**Time-box:** 2 evenings.
|
||||
|
||||
### Concepts
|
||||
|
||||
- Three levels of "as code", and you should know when each applies:
|
||||
1. **File provisioning** — YAML in `provisioning/datasources/` and
|
||||
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
|
||||
container. Grafana loads them at startup. Simplest, no state, no API tokens,
|
||||
perfect for a Compose demo. **This alone satisfies the requirement.**
|
||||
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
|
||||
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
|
||||
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
|
||||
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
|
||||
of Grafonnet (which is legacy and unofficial).
|
||||
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
|
||||
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
|
||||
when this lands on your Kubernetes cluster.
|
||||
- **Dashboards should answer questions, not display data.** A good service dashboard
|
||||
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
|
||||
is the pressure. Panels that nobody looks at during an incident are cost, not value.
|
||||
- **Alerts as code too.** Grafana alert rules, contact points and notification
|
||||
policies all provision from YAML. And keep the *alert definition* next to the
|
||||
dashboard that explains it.
|
||||
- **Instrument types.**
|
||||
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
|
||||
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
|
||||
connection-pool checkouts.
|
||||
- **Histogram** — a distribution. Request latency, payload size. This is what
|
||||
gives you percentiles.
|
||||
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
|
||||
read via a callback on collection (right for things you *poll*, like pool size).
|
||||
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
|
||||
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
|
||||
nonsense. Use histograms.
|
||||
- **Cardinality — the one thing that will bite you.** Every unique combination of
|
||||
metric name + label values is a separate time series in memory. Add `account_id`
|
||||
as a label on a 800-account demo and you've made 800 series; do it in production
|
||||
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
|
||||
~10 000 distinct values in production, it does not belong on a metric.** Put it on
|
||||
a span. Histograms multiply this (one series per bucket per label combination), so
|
||||
don't over-bucket either.
|
||||
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
|
||||
the concrete path), `http.status_code` class, `kind`, `currency`.
|
||||
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
|
||||
raw URL, timestamp, error message.
|
||||
- **RED and USE.**
|
||||
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
|
||||
per-service dashboard.
|
||||
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
|
||||
connection pools, queues, disk. Saturation is the one everyone forgets and the
|
||||
one that predicts collapse.
|
||||
- You want both: RED tells you the service is slow, USE tells you which resource
|
||||
is why.
|
||||
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
|
||||
and knowing both is the point:
|
||||
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
|
||||
`servicegraph` connectors read spans in flight and emit RED metrics and edge
|
||||
metrics. This is what most blog posts (including the OneUptime article) show.
|
||||
- *In the trace store:* Tempo's **metrics-generator** does the same job after
|
||||
ingestion, remote-writing the result to Prometheus. **This is what your platform
|
||||
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
|
||||
`send_exemplars: true`, remote-writing to a Prometheus started with
|
||||
`--web.enable-remote-write-receiver`.
|
||||
- So RED metrics, the service map and exemplars **already exist** before you write
|
||||
a single metric line. Do not add the collector connectors as well — you'd
|
||||
duplicate every series under a second set of names. **Go find the real metric
|
||||
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
|
||||
they will not match the article's.
|
||||
- The trade-off worth being able to argue: collector-side generation is cheaper
|
||||
and works before storage, but sees only what passes through that collector;
|
||||
Tempo-side generation sees everything that landed, survives collector restarts,
|
||||
and can be enabled per-tenant — at the cost of load on the trace store.
|
||||
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
|
||||
click a spike on a latency graph and land in the exact trace. On your platform
|
||||
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
|
||||
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
|
||||
`name: trace_id`. If you emit your own histograms with exemplars, the label must
|
||||
be `trace_id` to match.
|
||||
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
|
||||
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
|
||||
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
|
||||
the pull model is why Prometheus has service discovery and why `up` exists.
|
||||
|
||||
### Resources
|
||||
|
||||
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
|
||||
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
|
||||
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
|
||||
example.
|
||||
- Udemy backbone: dashboard design sections (panel types, transformations,
|
||||
variables, dashboard best practice).
|
||||
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
|
||||
Read the first half. Best cardinality explanation available.
|
||||
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
|
||||
not a fact.
|
||||
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
|
||||
`servicegraph` connector config and its RED PromQL queries almost verbatim.
|
||||
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
|
||||
|
||||
### Build
|
||||
|
||||
Your platform provisions **datasources only** — there is no dashboards provider at
|
||||
all. So this phase is entirely yours, and the right home for it is inside the
|
||||
Ansible role, using the same mechanism as everything else.
|
||||
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
|
||||
2. Custom application metrics, deliberately few:
|
||||
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
|
||||
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
|
||||
- `ledgerline.fraud.score` (Histogram; no labels)
|
||||
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
|
||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
||||
metrics, so pick a path and implement it:
|
||||
- *Platform grain:* expose `/metrics` on each service and add them to
|
||||
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
|
||||
playbook. Prometheus scrapes your services directly. Note the irony and the
|
||||
lesson — you're back to a pull model and a second client library, which is
|
||||
exactly the fragmentation OTel exists to fix.
|
||||
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
|
||||
Alloy so app metrics flow through the collector as the architecture drawing
|
||||
intends. This also closes a real diagram-vs-code gap in the repo.
|
||||
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
|
||||
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
|
||||
all three signals.
|
||||
Do the first to get moving; do one of the others afterwards and write up which
|
||||
you'd choose for a real platform team and why. That write-up is a Phase 10
|
||||
artefact.
|
||||
4. Deliberately break it: add `account_id` as a label to
|
||||
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
|
||||
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
|
||||
you'll never forget.
|
||||
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
|
||||
one. Understand why they differ slightly and when you'd want each.
|
||||
6. **Second migration of the call-counting assertions.** The three tests you moved
|
||||
to LogQL in Phase 4 can now count **client spans** in Tempo instead — a
|
||||
TraceQL query for `sanctions` client spans within a single trace answers
|
||||
"how many lookups did this one transaction make?" directly, which is the
|
||||
question the test was always really asking. Do it and compare all three
|
||||
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
|
||||
clearest possible demonstration of what each signal is actually for, on a
|
||||
problem you already understand.
|
||||
|
||||
1. **Extend the role.** Add
|
||||
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
|
||||
(a file-based dashboard provider pointing at a mounted directory), a
|
||||
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
|
||||
and copy them. The Compose template already mounts
|
||||
`./config/grafana/provisioning` read-only, so the provider is picked up on
|
||||
restart.
|
||||
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
|
||||
Foundation SDK (Python)**: one function producing a RED dashboard for a given
|
||||
service name, looped over the seven services; a second producing a system
|
||||
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
|
||||
the generator *and* its output; the Ansible role ships the output.
|
||||
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
|
||||
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
|
||||
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
|
||||
than a local Makefile target, because it's the same command that builds the
|
||||
platform.
|
||||
4. While you're in there, review the provisioned datasources and add what's missing:
|
||||
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
|
||||
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
|
||||
understand why `filterByTraceID: true` alone is sufficient here, and when it
|
||||
wouldn't be.
|
||||
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
|
||||
|
||||
**Done when:** re-running the playbook restores every dashboard from scratch, and no
|
||||
dashboard has ever been saved through the Grafana UI.
|
||||
**Done when:** you have a RED view per service, a working service map, and you can
|
||||
click a latency spike through to a trace via an exemplar.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user