Correct OTLP metrics/logs behaviour to 404, and start Phase 2

Alloy only registers an OTLP/HTTP route for a signal with an output, so
POST /v1/metrics and /v1/logs return 404 rather than being accepted and
silently dropped (verified live against otlp.vione.cloud). Fix the claim
everywhere it appeared. Also mark Phase 2 in progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-12 20:09:38 +03:30
co-authored by Claude Opus 5.5
parent 7719ef5fcf
commit 995c93982a
8 changed files with 29 additions and 18 deletions
+8 -3
View File
@@ -55,8 +55,12 @@ them against the live system rather than assuming — they were read from the co
at one point in time and the user may have since extended them.
1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
`traces` and nothing else. OTLP **metrics and logs are accepted and silently
discarded** — no error anywhere. This is deliberate: the platform's own diagram
`traces` and nothing else. OTLP **metrics and logs are rejected**: Alloy only
registers an OTLP/HTTP route for a signal that has an output, so
`POST /v1/metrics` and `POST /v1/logs` return **404** while `/v1/traces`
returns 200 (verified live 2026-10-12; gRPC is cluster-internal and untested).
The failure is visible only in the sender's own export-error logs — nothing
shows up in Alloy, Prometheus or Loki. This is deliberate: the platform's own diagram
labels the OTLP input "spans + resource attrs" and draws app metrics as a
separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
metrics "should be arriving"; check Prometheus or the Alloy UI first.
@@ -424,7 +428,8 @@ Things a well-meaning assistant does wrong here:
exporter pipelines. A correct-looking Collector config is a wrong answer here.
- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
already does this.
- **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in
- **Assuming OTLP metrics or logs arrive.** Alloy 404s both over OTLP/HTTP, and
the only trace of that is an export error in the app's logs. Verify in
Prometheus / Loki before claiming anything works.
- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
Loki derived field with no visible error.
+2 -1
View File
@@ -263,7 +263,8 @@ Three constraints inherited from that platform, worth knowing before you debug
anything:
- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
accepted and silently dropped — by design; the platform's diagram labels that
rejected with a 404 on `/v1/metrics` and `/v1/logs`, and the only sign is an
export error in the sending app's logs. This is by design; the platform's diagram labels that
input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
+6 -4
View File
@@ -231,10 +231,12 @@ otelcol.receiver.otlp "default" {
}
```
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
your app sends are accepted and then **silently discarded** — no error, no warning,
just nothing arriving in Grafana. This is the single most likely thing to eat an
evening.
There is **no `metrics` output and no `logs` output**, and Alloy only registers
an OTLP/HTTP route for a signal that has one. So OTLP metrics and OTLP logs your
app sends get a **404** from `/v1/metrics` and `/v1/logs` (verified live
2026-10-12). The only evidence is an export error in your app's own logs, which
is easy to miss. Nothing shows up in Alloy, Prometheus or Loki; data just never
arrives in Grafana. This is the single most likely thing to eat an evening.
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
+4 -3
View File
@@ -2,10 +2,11 @@
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
**Status:** Not started
**Status:** In progress
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
described above. Waiting to begin.
**Message:** Started 2026-10-12. Step 1 (deploy the platform) is done and
verified live, apart from reading the Alloy component graph. Next: step 2,
zero-code auto-instrumentation of `sanctions`.
**Time-box:** one evening.
+2 -2
View File
@@ -94,8 +94,8 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
- `ledgerline.fraud.score` (Histogram; no labels)
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
metrics, so pick a path and implement it:
3. **Get your custom metrics somewhere they'll actually land.** Alloy rejects OTLP
metrics (404 on `/v1/metrics`), so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
+3 -3
View File
@@ -17,7 +17,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
| 2 | [`phase-2.md`](phase-2.md) | In progress | Concepts + first trace | 1 evening | One trace in Tempo |
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
@@ -47,8 +47,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
in both directions.
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
silently. Tempo's metrics-generator already produces span metrics and the
limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics
(404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the
service graph — don't duplicate it in the collector.
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
+2 -1
View File
@@ -43,7 +43,8 @@ OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.
empty in the role too; see `prometheus/prometheus.yml`).
The CLAUDE.md platform facts still hold:
- OTLP metrics and logs are silently dropped.
- OTLP metrics and logs are rejected: Alloy returns a 404 on `/v1/metrics` and
`/v1/logs`, so the only sign is an export error in the app's own logs.
- Span metrics and the service graph come from Tempo's metrics-generator.
- The Loki derived field matches lowercase `trace_id`.
- Datasources are provisioned; dashboards are not.
+2 -1
View File
@@ -7,7 +7,8 @@
// observability/alloy-agent/config.alloy.
//
// What remains is unchanged: the OTLP receiver wires a `traces` output and
// nothing else. OTLP metrics and logs are accepted and silently dropped.
// nothing else. The HTTP server only registers routes for signals with an
// output, so OTLP/HTTP POSTs to /v1/metrics and /v1/logs get a 404.
logging {
level = "info"