From 88700c000d59635401f740dca00c827d4bc89866 Mon Sep 17 00:00:00 2001 From: Movses Aslanyan Date: Sat, 24 Oct 2026 22:40:58 +0330 Subject: [PATCH] Label Tempo's exemplars trace_id so they link to the trace Tempo's metrics-generator defaults the exemplar label to traceID, which the Prometheus datasource's exemplarTraceIdDestinations (trace_id) never matched: exemplars rendered in Grafana but didn't link to Tempo. Set trace_id_label_name in the overrides so trace_id is the one correlation key everywhere. Co-Authored-By: Claude Opus 5.5 --- k8s/README.md | 4 ++++ k8s/tempo/tempo.yml | 5 +++++ 2 files changed, 9 insertions(+) diff --git a/k8s/README.md b/k8s/README.md index c2d66ea..371d781 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -45,6 +45,10 @@ OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP. Traefik routes to that container's `:9464` behind basic auth. The scrape credentials are the `prometheus-scrape-auth` secret, mounted at `/etc/prometheus/secrets`. +- **Tempo labels exemplars `trace_id`** (`trace_id_label_name` in the + overrides). Tempo's default is `traceID`, which the Prometheus datasource's + `exemplarTraceIdDestinations` (`trace_id`) doesn't match, so span-metric + exemplars rendered in Grafana but didn't link to the trace. The CLAUDE.md platform facts still hold: - OTLP metrics and logs are rejected: Alloy returns a 404 on `/v1/metrics` and diff --git a/k8s/tempo/tempo.yml b/k8s/tempo/tempo.yml index 1ee5f01..4fab6c8 100644 --- a/k8s/tempo/tempo.yml +++ b/k8s/tempo/tempo.yml @@ -45,3 +45,8 @@ overrides: defaults: metrics_generator: processors: [service-graphs, span-metrics, local-blocks] + # Not in the Ansible template. Without it the generator labels exemplars + # traceID (the code default; the docs wrongly say trace_id), which the + # Prometheus datasource's exemplarTraceIdDestinations (trace_id) never + # matches, so exemplars render but don't link to Tempo. + trace_id_label_name: trace_id