Correct OTLP metrics/logs behaviour to 404, and start Phase 2
Alloy only registers an OTLP/HTTP route for a signal with an output, so POST /v1/metrics and /v1/logs return 404 rather than being accepted and silently dropped (verified live against otlp.vione.cloud). Fix the claim everywhere it appeared. Also mark Phase 2 in progress. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
7719ef5fcf
commit
995c93982a
@@ -231,10 +231,12 @@ otelcol.receiver.otlp "default" {
|
||||
}
|
||||
```
|
||||
|
||||
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
|
||||
your app sends are accepted and then **silently discarded** — no error, no warning,
|
||||
just nothing arriving in Grafana. This is the single most likely thing to eat an
|
||||
evening.
|
||||
There is **no `metrics` output and no `logs` output**, and Alloy only registers
|
||||
an OTLP/HTTP route for a signal that has one. So OTLP metrics and OTLP logs your
|
||||
app sends get a **404** from `/v1/metrics` and `/v1/logs` (verified live
|
||||
2026-10-12). The only evidence is an export error in your app's own logs, which
|
||||
is easy to miss. Nothing shows up in Alloy, Prometheus or Loki; data just never
|
||||
arrives in Grafana. This is the single most likely thing to eat an evening.
|
||||
|
||||
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
|
||||
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
|
||||
|
||||
@@ -2,10 +2,11 @@
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||||
|
||||
**Status:** Not started
|
||||
**Status:** In progress
|
||||
|
||||
**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is
|
||||
described above. Waiting to begin.
|
||||
**Message:** Started 2026-10-12. Step 1 (deploy the platform) is done and
|
||||
verified live, apart from reading the Alloy component graph. Next: step 2,
|
||||
zero-code auto-instrumentation of `sanctions`.
|
||||
|
||||
**Time-box:** one evening.
|
||||
|
||||
|
||||
@@ -94,8 +94,8 @@ first), but sequenced after Phase 4 per the roadmap's ordering.
|
||||
- `ledgerline.fraud.score` (Histogram; no labels)
|
||||
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
|
||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
||||
metrics, so pick a path and implement it:
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy rejects OTLP
|
||||
metrics (404 on `/v1/metrics`), so pick a path and implement it:
|
||||
- *Platform grain:* expose `/metrics` on each service and add scrape jobs to
|
||||
`k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`),
|
||||
then `kubectl apply -k k8s/`. Prometheus scrapes your services directly.
|
||||
|
||||
@@ -17,7 +17,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
|
||||
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
|
||||
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
|
||||
| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
|
||||
| 2 | [`phase-2.md`](phase-2.md) | In progress | Concepts + first trace | 1 evening | One trace in Tempo |
|
||||
| 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
||||
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
||||
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
|
||||
@@ -47,8 +47,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
|
||||
in both directions.
|
||||
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
|
||||
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
|
||||
silently. Tempo's metrics-generator already produces span metrics and the
|
||||
limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics
|
||||
(404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the
|
||||
service graph — don't duplicate it in the collector.
|
||||
- **6 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
||||
Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so
|
||||
|
||||
Reference in New Issue
Block a user