diff --git a/CLAUDE.md b/CLAUDE.md index 743ed8f..eb81ff1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -55,8 +55,12 @@ them against the live system rather than assuming — they were read from the co at one point in time and the user may have since extended them. 1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares - `traces` and nothing else. OTLP **metrics and logs are accepted and silently - discarded** — no error anywhere. This is deliberate: the platform's own diagram + `traces` and nothing else. OTLP **metrics and logs are rejected**: Alloy only + registers an OTLP/HTTP route for a signal that has an output, so + `POST /v1/metrics` and `POST /v1/logs` return **404** while `/v1/traces` + returns 200 (verified live 2026-10-12; gRPC is cluster-internal and untested). + The failure is visible only in the sender's own export-error logs — nothing + shows up in Alloy, Prometheus or Loki. This is deliberate: the platform's own diagram labels the OTLP input "spans + resource attrs" and draws app metrics as a separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP metrics "should be arriving"; check Prometheus or the Alloy UI first. @@ -424,7 +428,8 @@ Things a well-meaning assistant does wrong here: exporter pipelines. A correct-looking Collector config is a wrong answer here. - **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator already does this. -- **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in +- **Assuming OTLP metrics or logs arrive.** Alloy 404s both over OTLP/HTTP, and + the only trace of that is an export error in the app's logs. Verify in Prometheus / Loki before claiming anything works. - **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned Loki derived field with no visible error. diff --git a/README.md b/README.md index 06517a2..d52f7dd 100644 --- a/README.md +++ b/README.md @@ -263,7 +263,8 @@ Three constraints inherited from that platform, worth knowing before you debug anything: - **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are - accepted and silently dropped — by design; the platform's diagram labels that + rejected with a 404 on `/v1/metrics` and `/v1/logs`, and the only sign is an + export error in the sending app's logs. This is by design; the platform's diagram labels that input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`. - **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`, diff --git a/docs/roadmap/intro.md b/docs/roadmap/intro.md index 8eee340..1d71b7c 100644 --- a/docs/roadmap/intro.md +++ b/docs/roadmap/intro.md @@ -231,10 +231,12 @@ otelcol.receiver.otlp "default" { } ``` -There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs -your app sends are accepted and then **silently discarded** — no error, no warning, -just nothing arriving in Grafana. This is the single most likely thing to eat an -evening. +There is **no `metrics` output and no `logs` output**, and Alloy only registers +an OTLP/HTTP route for a signal that has one. So OTLP metrics and OTLP logs your +app sends get a **404** from `/v1/metrics` and `/v1/logs` (verified live +2026-10-12). The only evidence is an export error in your app's own logs, which +is easy to miss. Nothing shows up in Alloy, Prometheus or Loki; data just never +arrives in Grafana. This is the single most likely thing to eat an evening. The diagram confirms this is deliberate: the OTLP input is labelled "spans + resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition diff --git a/docs/roadmap/phase-2.md b/docs/roadmap/phase-2.md index 6cf7329..a9c4380 100644 --- a/docs/roadmap/phase-2.md +++ b/docs/roadmap/phase-2.md @@ -2,10 +2,11 @@ *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)* -**Status:** Not started +**Status:** In progress -**Message:** Blocked on nothing — Phases 0 and 1 are done and the platform is -described above. Waiting to begin. +**Message:** Started 2026-10-12. Step 1 (deploy the platform) is done and +verified live, apart from reading the Alloy component graph. Next: step 2, +zero-code auto-instrumentation of `sanctions`. **Time-box:** one evening. diff --git a/docs/roadmap/phase-5.md b/docs/roadmap/phase-5.md index 0abf5b4..fc74238 100644 --- a/docs/roadmap/phase-5.md +++ b/docs/roadmap/phase-5.md @@ -94,8 +94,8 @@ first), but sequenced after Phase 4 per the roadmap's ordering. - `ledgerline.fraud.score` (Histogram; no labels) - `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`) - `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`) -3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP - metrics, so pick a path and implement it: +3. **Get your custom metrics somewhere they'll actually land.** Alloy rejects OTLP + metrics (404 on `/v1/metrics`), so pick a path and implement it: - *Platform grain:* expose `/metrics` on each service and add scrape jobs to `k8s/prometheus/prometheus.yml` (the role's `prometheus_extra_scrape_jobs`), then `kubectl apply -k k8s/`. Prometheus scrapes your services directly. diff --git a/docs/roadmap/summary.md b/docs/roadmap/summary.md index c71dc15..0e2cc7f 100644 --- a/docs/roadmap/summary.md +++ b/docs/roadmap/summary.md @@ -17,7 +17,7 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. | 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` | | 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations | | 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` | -| 2 | [`phase-2.md`](phase-2.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo | +| 2 | [`phase-2.md`](phase-2.md) | In progress | Concepts + first trace | 1 evening | One trace in Tempo | | 3 | [`phase-3.md`](phase-3.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | | 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation | | 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars | @@ -47,8 +47,8 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. structured events carrying a lowercase `trace_id`, so Loki and Tempo link in both directions. - **5 — Metrics.** Instrument RED metrics per service, understand cardinality - limits, and route app metrics to Prometheus given Alloy drops OTLP metrics - silently. Tempo's metrics-generator already produces span metrics and the + limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics + (404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the service graph — don't duplicate it in the collector. - **6 — Dashboards and alerts as code.** Generate dashboard JSON with the Grafana Foundation SDK and ship it with the platform (`k8s/grafana/`), so diff --git a/k8s/README.md b/k8s/README.md index e42b8f6..5834549 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -43,7 +43,8 @@ OTLP/gRPC (4317) is cluster-internal only. Apps use OTLP/HTTP. empty in the role too; see `prometheus/prometheus.yml`). The CLAUDE.md platform facts still hold: -- OTLP metrics and logs are silently dropped. +- OTLP metrics and logs are rejected: Alloy returns a 404 on `/v1/metrics` and + `/v1/logs`, so the only sign is an export error in the app's own logs. - Span metrics and the service graph come from Tempo's metrics-generator. - The Loki derived field matches lowercase `trace_id`. - Datasources are provisioned; dashboards are not. diff --git a/k8s/alloy/config.alloy b/k8s/alloy/config.alloy index 5e1745e..f7f5174 100644 --- a/k8s/alloy/config.alloy +++ b/k8s/alloy/config.alloy @@ -7,7 +7,8 @@ // observability/alloy-agent/config.alloy. // // What remains is unchanged: the OTLP receiver wires a `traces` output and -// nothing else. OTLP metrics and logs are accepted and silently dropped. +// nothing else. The HTTP server only registers routes for signals with an +// output, so OTLP/HTTP POSTs to /v1/metrics and /v1/logs get a 404. logging { level = "info"