Add dashboards and alerts as code, mark Phase 6 done

Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK)
writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service
and a system overview -- shipped through a file provider and
configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by
deleting Grafana and its volume and re-applying. Adds tracesToMetrics to
the Tempo datasource.

Alerts: Prometheus rule files delivering to Alertmanager, under a
labels/annotations contract enforced by observability/alerts/check_contract.py,
with runbooks in docs/runbooks/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-25 20:01:52 +03:30
co-authored by Claude Sonnet 5.5
parent b50f8c3b24
commit fef735c922
19 changed files with 1314 additions and 7 deletions
+16
View File
@@ -38,6 +38,22 @@ datasources:
filterByTraceID: true
spanStartTimeShift: -1m
spanEndTimeShift: 1m
# QUESTION (answer in your own words; verify by reading the LogQL that
# "Logs for this span" generates): why is `filterByTraceID: true` enough
# with no `tags` mapping here, and when would it not be?
tracesToMetrics:
datasourceUid: prometheus
# Wider than the span: rate() needs a few scrape intervals of data.
spanStartTimeShift: -2m
spanEndTimeShift: 2m
# span attribute -> Prometheus label on traces_spanmetrics_*
tags:
- key: service.name
value: service
queries:
# `$$` is the provisioning-file escape for a literal `$`.
- name: Request rate
query: sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))
serviceMap:
datasourceUid: prometheus
nodeGraph: