Correct OTLP metrics/logs behaviour to 404, and start Phase 2

Alloy only registers an OTLP/HTTP route for a signal with an output, so
POST /v1/metrics and /v1/logs return 404 rather than being accepted and
silently dropped (verified live against otlp.vione.cloud). Fix the claim
everywhere it appeared. Also mark Phase 2 in progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 20:09:38 +03:30
co-authored by Claude Opus 5.5
parent 7719ef5fcf
commit 995c93982a
8 changed files with 29 additions and 18 deletions
+8 -3
View File
@@ -55,8 +55,12 @@ them against the live system rather than assuming — they were read from the co
at one point in time and the user may have since extended them. at one point in time and the user may have since extended them.
1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares 1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
`traces` and nothing else. OTLP **metrics and logs are accepted and silently `traces` and nothing else. OTLP **metrics and logs are rejected**: Alloy only
discarded** — no error anywhere. This is deliberate: the platform's own diagram registers an OTLP/HTTP route for a signal that has an output, so
`POST /v1/metrics` and `POST /v1/logs` return **404** while `/v1/traces`
returns 200 (verified live 2026-10-12; gRPC is cluster-internal and untested).
The failure is visible only in the sender's own export-error logs — nothing
shows up in Alloy, Prometheus or Loki. This is deliberate: the platform's own diagram
labels the OTLP input "spans + resource attrs" and draws app metrics as a labels the OTLP input "spans + resource attrs" and draws app metrics as a
separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
metrics "should be arriving"; check Prometheus or the Alloy UI first. metrics "should be arriving"; check Prometheus or the Alloy UI first.
@@ -424,7 +428,8 @@ Things a well-meaning assistant does wrong here:
exporter pipelines. A correct-looking Collector config is a wrong answer here. exporter pipelines. A correct-looking Collector config is a wrong answer here.
- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator - **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
already does this. already does this.
- **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in - **Assuming OTLP metrics or logs arrive.** Alloy 404s both over OTLP/HTTP, and
the only trace of that is an export error in the app's logs. Verify in
Prometheus / Loki before claiming anything works. Prometheus / Loki before claiming anything works.
- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned - **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
Loki derived field with no visible error. Loki derived field with no visible error.
+2 -1
View File
@@ -263,7 +263,8 @@ Three constraints inherited from that platform, worth knowing before you debug
anything: anything:
- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are - **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
accepted and silently dropped — by design; the platform's diagram labels that rejected with a 404 on `/v1/metrics` and `/v1/logs`, and the only sign is an
export error in the sending app's logs. This is by design; the platform's diagram labels that
input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`. each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`, - **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
+6 -4
View File
@@ -231,10 +231,12 @@ otelcol.receiver.otlp "default" {
} }
``` ```
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs There is **no `metrics` output and no `logs` output**, and Alloy only registers
your app sends are accepted and then **silently discarded** — no error, no warning, an OTLP/HTTP route for a signal that has one. So OTLP metrics and OTLP logs your
just nothing arriving in Grafana. This is the single most likely thing to eat an app sends get a **404** from `/v1/metrics` and `/v1/logs` (verified live
evening. 2026-10-12). The only evidence is an export error in your app's own logs, which
is easy to miss. Nothing shows up in Alloy, Prometheus or Loki; data just never
arrives in Grafana. This is the single most likely thing to eat an evening.
The diagram confirms this is deliberate: the OTLP input is labelled "spans + The diagram confirms this is deliberate: the OTLP input is labelled "spans +
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
+4 -3
View File
@@ -2,10 +2,11 @@
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)* *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
**Status:** Not started **Status:** In progress
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is **Message:** Started 2026-10-12. Step 1 (deploy the platform) is done and
described above. Waiting to begin. verified live, apart from reading the Alloy component graph. Next: step 2,
zero-code auto-instrumentation of `sanctions`.
**Time-box:** one evening. **Time-box:** one evening.
+2 -2
View File
@@ -94,8 +94,8 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
- `ledgerline.fraud.score` (Histogram; no labels) - `ledgerline.fraud.score` (Histogram; no labels)
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`) - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`) - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP 3. **Get your custom metrics somewhere they'll actually land.** Alloy rejects OTLP
metrics, so pick a path and implement it: metrics (404 on `/v1/metrics`), so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to - *Platform grain:* expose `/metrics` on each service and add scrape jobs to
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`), `k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly. then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
+3 -3
View File
@@ -17,7 +17,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` | | 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations | | 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` | | 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo | | 2 | [`phase-2.md`](phase-2.md) | In progress | Concepts + first trace | 1 evening | One trace in Tempo |
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | | 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation | | 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars | | 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
@@ -47,8 +47,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
structured events carrying a lowercase `trace_id`, so Loki and Tempo link structured events carrying a lowercase `trace_id`, so Loki and Tempo link
in both directions. in both directions.
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality - **5 — Metrics.** Instrument RED metrics per service, understand cardinality
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics
silently. Tempo's metrics-generator already produces span metrics and the (404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the
service graph — don't duplicate it in the collector. service graph — don't duplicate it in the collector.
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the - **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
+2 -1
View File
@@ -43,7 +43,8 @@ OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.
empty in the role too; see `prometheus/prometheus.yml`). empty in the role too; see `prometheus/prometheus.yml`).
The CLAUDE.md platform facts still hold: The CLAUDE.md platform facts still hold:
- OTLP metrics and logs are silently dropped. - OTLP metrics and logs are rejected: Alloy returns a 404 on `/v1/metrics` and
`/v1/logs`, so the only sign is an export error in the app's own logs.
- Span metrics and the service graph come from Tempo's metrics-generator. - Span metrics and the service graph come from Tempo's metrics-generator.
- The Loki derived field matches lowercase `trace_id`. - The Loki derived field matches lowercase `trace_id`.
- Datasources are provisioned; dashboards are not. - Datasources are provisioned; dashboards are not.
+2 -1
View File
@@ -7,7 +7,8 @@
// observability/alloy-agent/config.alloy. // observability/alloy-agent/config.alloy.
// //
// What remains is unchanged: the OTLP receiver wires a `traces` output and // What remains is unchanged: the OTLP receiver wires a `traces` output and
// nothing else. OTLP metrics and logs are accepted and silently dropped. // nothing else. The HTTP server only registers routes for signals with an
// output, so OTLP/HTTP POSTs to /v1/metrics and /v1/logs get a 404.
logging { logging {
level = "info" level = "info"