Add dashboards and alerts as code, mark Phase 6 done
Dashboards: observability/dashboards/generate.py (Grafana Foundation SDK) writes k8s/grafana/dashboards/*.json -- a RED dashboard per HTTP service and a system overview -- shipped through a file provider and configMapGenerators, so `kubectl apply -k k8s/` rebuilds them. Verified by deleting Grafana and its volume and re-applying. Adds tracesToMetrics to the Tempo datasource. Alerts: Prometheus rule files delivering to Alertmanager, under a labels/annotations contract enforced by observability/alerts/check_contract.py, with runbooks in docs/runbooks/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5.5
parent
b50f8c3b24
commit
fef735c922
@@ -38,6 +38,22 @@ datasources:
|
||||
filterByTraceID: true
|
||||
spanStartTimeShift: -1m
|
||||
spanEndTimeShift: 1m
|
||||
# QUESTION (answer in your own words; verify by reading the LogQL that
|
||||
# "Logs for this span" generates): why is `filterByTraceID: true` enough
|
||||
# with no `tags` mapping here, and when would it not be?
|
||||
tracesToMetrics:
|
||||
datasourceUid: prometheus
|
||||
# Wider than the span: rate() needs a few scrape intervals of data.
|
||||
spanStartTimeShift: -2m
|
||||
spanEndTimeShift: 2m
|
||||
# span attribute -> Prometheus label on traces_spanmetrics_*
|
||||
tags:
|
||||
- key: service.name
|
||||
value: service
|
||||
queries:
|
||||
# `$$` is the provisioning-file escape for a literal `$`.
|
||||
- name: Request rate
|
||||
query: sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
nodeGraph:
|
||||
|
||||
Reference in New Issue
Block a user