Correct OTLP metrics/logs behaviour to 404, and start Phase 2
Alloy only registers an OTLP/HTTP route for a signal with an output, so POST /v1/metrics and /v1/logs return 404 rather than being accepted and silently dropped (verified live against otlp.vione.cloud). Fix the claim everywhere it appeared. Also mark Phase 2 in progress. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
7719ef5fcf
commit
995c93982a
@@ -55,8 +55,12 @@ them against the live system rather than assuming — they were read from the co
|
|||||||
at one point in time and the user may have since extended them.
|
at one point in time and the user may have since extended them.
|
||||||
|
|
||||||
1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
|
1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
|
||||||
`traces` and nothing else. OTLP **metrics and logs are accepted and silently
|
`traces` and nothing else. OTLP **metrics and logs are rejected**: Alloy only
|
||||||
discarded** — no error anywhere. This is deliberate: the platform's own diagram
|
registers an OTLP/HTTP route for a signal that has an output, so
|
||||||
|
`POST /v1/metrics` and `POST /v1/logs` return **404** while `/v1/traces`
|
||||||
|
returns 200 (verified live 2026-10-12; gRPC is cluster-internal and untested).
|
||||||
|
The failure is visible only in the sender's own export-error logs — nothing
|
||||||
|
shows up in Alloy, Prometheus or Loki. This is deliberate: the platform's own diagram
|
||||||
labels the OTLP input "spans + resource attrs" and draws app metrics as a
|
labels the OTLP input "spans + resource attrs" and draws app metrics as a
|
||||||
separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
|
separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
|
||||||
metrics "should be arriving"; check Prometheus or the Alloy UI first.
|
metrics "should be arriving"; check Prometheus or the Alloy UI first.
|
||||||
@@ -424,7 +428,8 @@ Things a well-meaning assistant does wrong here:
|
|||||||
exporter pipelines. A correct-looking Collector config is a wrong answer here.
|
exporter pipelines. A correct-looking Collector config is a wrong answer here.
|
||||||
- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
|
- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
|
||||||
already does this.
|
already does this.
|
||||||
- **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in
|
- **Assuming OTLP metrics or logs arrive.** Alloy 404s both over OTLP/HTTP, and
|
||||||
|
the only trace of that is an export error in the app's logs. Verify in
|
||||||
Prometheus / Loki before claiming anything works.
|
Prometheus / Loki before claiming anything works.
|
||||||
- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
|
- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
|
||||||
Loki derived field with no visible error.
|
Loki derived field with no visible error.
|
||||||
|
|||||||
@@ -263,7 +263,8 @@ Three constraints inherited from that platform, worth knowing before you debug
|
|||||||
anything:
|
anything:
|
||||||
|
|
||||||
- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
|
- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
|
||||||
accepted and silently dropped — by design; the platform's diagram labels that
|
rejected with a 404 on `/v1/metrics` and `/v1/logs`, and the only sign is an
|
||||||
|
export error in the sending app's logs. This is by design; the platform's diagram labels that
|
||||||
input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
|
input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
|
||||||
each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
|
each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
|
||||||
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
|
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
|
||||||
|
|||||||
@@ -231,10 +231,12 @@ otelcol.receiver.otlp "default" {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
|
There is **no `metrics` output and no `logs` output**, and Alloy only registers
|
||||||
your app sends are accepted and then **silently discarded** — no error, no warning,
|
an OTLP/HTTP route for a signal that has one. So OTLP metrics and OTLP logs your
|
||||||
just nothing arriving in Grafana. This is the single most likely thing to eat an
|
app sends get a **404** from `/v1/metrics` and `/v1/logs` (verified live
|
||||||
evening.
|
2026-10-12). The only evidence is an export error in your app's own logs, which
|
||||||
|
is easy to miss. Nothing shows up in Alloy, Prometheus or Loki; data just never
|
||||||
|
arrives in Grafana. This is the single most likely thing to eat an evening.
|
||||||
|
|
||||||
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
|
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
|
||||||
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
|
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
|
||||||
|
|||||||
@@ -2,10 +2,11 @@
|
|||||||
|
|
||||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||||||
|
|
||||||
**Status:** Not started
|
**Status:** In progress
|
||||||
|
|
||||||
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
|
**Message:** Started 2026-10-12. Step 1 (deploy the platform) is done and
|
||||||
described above. Waiting to begin.
|
verified live, apart from reading the Alloy component graph. Next: step 2,
|
||||||
|
zero-code auto-instrumentation of `sanctions`.
|
||||||
|
|
||||||
**Time-box:** one evening.
|
**Time-box:** one evening.
|
||||||
|
|
||||||
|
|||||||
@@ -94,8 +94,8 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
|
|||||||
- `ledgerline.fraud.score` (Histogram; no labels)
|
- `ledgerline.fraud.score` (Histogram; no labels)
|
||||||
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
|
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
|
||||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
3. **Get your custom metrics somewhere they'll actually land.** Alloy rejects OTLP
|
||||||
metrics, so pick a path and implement it:
|
metrics (404 on `/v1/metrics`), so pick a path and implement it:
|
||||||
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
|
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
|
||||||
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
|
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
|
||||||
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
|
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
|||||||
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
|
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
|
||||||
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
|
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
|
||||||
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
|
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
|
||||||
| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
|
| 2 | [`phase-2.md`](phase-2.md) | In progress | Concepts + first trace | 1 evening | One trace in Tempo |
|
||||||
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
||||||
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
||||||
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
|
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
|
||||||
@@ -47,8 +47,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
|||||||
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
|
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
|
||||||
in both directions.
|
in both directions.
|
||||||
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
|
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
|
||||||
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
|
limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics
|
||||||
silently. Tempo's metrics-generator already produces span metrics and the
|
(404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the
|
||||||
service graph — don't duplicate it in the collector.
|
service graph — don't duplicate it in the collector.
|
||||||
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
||||||
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
|
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
|
||||||
|
|||||||
+2
-1
@@ -43,7 +43,8 @@ OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP.
|
|||||||
empty in the role too; see `prometheus/prometheus.yml`).
|
empty in the role too; see `prometheus/prometheus.yml`).
|
||||||
|
|
||||||
The CLAUDE.md platform facts still hold:
|
The CLAUDE.md platform facts still hold:
|
||||||
- OTLP metrics and logs are silently dropped.
|
- OTLP metrics and logs are rejected: Alloy returns a 404 on `/v1/metrics` and
|
||||||
|
`/v1/logs`, so the only sign is an export error in the app's own logs.
|
||||||
- Span metrics and the service graph come from Tempo's metrics-generator.
|
- Span metrics and the service graph come from Tempo's metrics-generator.
|
||||||
- The Loki derived field matches lowercase `trace_id`.
|
- The Loki derived field matches lowercase `trace_id`.
|
||||||
- Datasources are provisioned; dashboards are not.
|
- Datasources are provisioned; dashboards are not.
|
||||||
|
|||||||
@@ -7,7 +7,8 @@
|
|||||||
// observability/alloy-agent/config.alloy.
|
// observability/alloy-agent/config.alloy.
|
||||||
//
|
//
|
||||||
// What remains is unchanged: the OTLP receiver wires a `traces` output and
|
// What remains is unchanged: the OTLP receiver wires a `traces` output and
|
||||||
// nothing else. OTLP metrics and logs are accepted and silently dropped.
|
// nothing else. The HTTP server only registers routes for signals with an
|
||||||
|
// output, so OTLP/HTTP POSTs to /v1/metrics and /v1/logs get a 404.
|
||||||
|
|
||||||
logging {
|
logging {
|
||||||
level = "info"
|
level = "info"
|
||||||
|
|||||||
Reference in New Issue
Block a user