From 44d780161714befe2b5e15b90f4c2a738220fae1 Mon Sep 17 00:00:00 2001 From: Movses Aslanyan Date: Tue, 13 Oct 2026 21:32:43 +0330 Subject: [PATCH] Mark Phases 3 and 4 done, and start Phase 5 Phase 4 records what was built, the two skipped Build steps (OTLP-logs experiment, LogQL test migration) and what carries forward. Phase 3 records its outcome, with the chaos-nightmare and blocking_event_loop criteria attested by the user. Co-Authored-By: Claude Opus 5.5 --- docs/roadmap/phase-3.md | 18 ++++++++++++++---- docs/roadmap/phase-4.md | 40 +++++++++++++++++++++++++++++++++++++--- docs/roadmap/phase-5.md | 5 ++--- docs/roadmap/summary.md | 12 +++++++----- 4 files changed, 60 insertions(+), 15 deletions(-) diff --git a/docs/roadmap/phase-3.md b/docs/roadmap/phase-3.md index d112886..80401cd 100644 --- a/docs/roadmap/phase-3.md +++ b/docs/roadmap/phase-3.md @@ -2,11 +2,21 @@ *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)* -**Status:** In progress +**Status:** Done -**Message:** Started 2026-10-12. Phase 2 got `sanctions` tracing through zero-code -`opentelemetry-instrument`, so no `common/telemetry.py` exists yet. Build step 1 -writes that bootstrap by hand, in the code, to replace the wrapper. +**Message:** Started 2026-10-12, done 2026-10-13. +- `common/telemetry.py` bootstraps tracing in code for all seven services, + replacing Phase 2's zero-code wrapper. It covers FastAPI, SQLAlchemy, httpx + and Redis instrumentation. +- Trace context crosses the Redis Stream by hand: it's injected into the message + inside a PRODUCER span and extracted by both consumer groups. +- Business spans were added, swallowed exceptions are recorded on their spans, + and the response's `request_id` is now the trace ID. The guard suite is + retired. +- One request produces one connected trace across all seven services, both + consumers included (verified live during Phase 4: 51 spans, 7 services). +- The chaos-nightmare and "explain `blocking_event_loop`" criteria were attested + by the user. **Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task. diff --git a/docs/roadmap/phase-4.md b/docs/roadmap/phase-4.md index 476d4c0..f04c58f 100644 --- a/docs/roadmap/phase-4.md +++ b/docs/roadmap/phase-4.md @@ -2,10 +2,44 @@ *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)* -**Status:** Not started +**Status:** Done -**Message:** Depends on Phase 3 (trace IDs must exist before log correlation -means anything). +**Message:** Done 2026-10-13. Both directions work in two clicks: a Loki line's +`TraceID` derived field opens the trace, and a Tempo span's logs link returns that +request's lines from every service that logged, the audit/notifier consumers +included (verified live on trace `a4f034b1d66a69b357aef66a647afcef`). + +- `common/logging_config.py`: structlog → one JSON object per line on stdout, with + stdlib and uvicorn records going through the same processor chain. Processors + add level, logger, ISO-8601 UTC timestamp, `service`, lowercase `trace_id`/`span_id` + (only inside a valid span) and field-name redaction. `uvicorn.access` is left + alone on purpose, because re-enabling its propagation undoes `--no-access-log`. +- Every call site is now a past-tense event with shared field names (`account_id`, + `transaction_id`, `amount_minor`, `risk_score`, `status_code`, `consumer_group`, + `message_id`, `chaos_flag`). New events: `transaction_failed` (the gateway's 502 + path used to log nothing), `transaction_scored`, `holder_screened`, + `stream_message_processed`. `holder_name` is no longer logged at all. +- The czy agent sets `service_name` from the Compose service name. Before that + change, Loki derived it from `container` (`ledgerline-sanctions-1`), so Grafana's + generated `{service_name="sanctions"}` matched nothing. Streams written before + 2026-10-13 17:52 UTC keep the old value and don't link. +- **Skipped:** Build step 5 (the OTLP-logs silent-drop experiment; Alloy still has no + `logs` output) and the second Build step 6 (migrating `count_log_lines` to LogQL). The tests run + against the local stack, whose logs never reach Loki, so the migration first + needs a local log path into Loki. +- **Carry forward:** + - The pathology tests still count `docker compose logs` lines matching + `(?i)lookup` (sanctions) and `(?i)healthz` (downstreams). Those words must stay + in exactly `lookup_received` and `healthz_probe_received`. + - `chaos_engaged` lines stay at INFO by choice, so they're visible in Loki during + Phase 9. + - httpx's `HTTP Request:` INFO lines are kept; most of gateway's volume is the + frontend health poll. + - The deploy hot-reloads bind-mounted code into the old image before + `--build`, so a commit that adds a dependency crash-loops the services for + about a minute. + - `alloy-agent`'s config is a single-file bind mount: after `git pull` it needs + `docker compose ... restart alloy-agent` to see the new file. **Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win. diff --git a/docs/roadmap/phase-5.md b/docs/roadmap/phase-5.md index fc74238..0dd2ad3 100644 --- a/docs/roadmap/phase-5.md +++ b/docs/roadmap/phase-5.md @@ -2,10 +2,9 @@ *Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)* -**Status:** Not started +**Status:** In progress -**Message:** Not blocked technically (metrics don't strictly need tracing -first), but sequenced after Phase 4 per the roadmap's ordering. +**Message:** Started 2026-10-13. **Time-box:** 2 evenings. diff --git a/docs/roadmap/summary.md b/docs/roadmap/summary.md index 7fadfe6..299b4cb 100644 --- a/docs/roadmap/summary.md +++ b/docs/roadmap/summary.md @@ -18,9 +18,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. | 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations | | 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` | | 2 | [`phase-2.md`](phase-2.md) | Done | Concepts + first trace | 1 evening | One trace in Tempo | -| 3 | [`phase-3.md`](phase-3.md) | In progress | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | -| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation | -| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars | +| 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop | +| 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation | +| 5 | [`phase-5.md`](phase-5.md) | In progress | Metrics | 2 evenings | RED per service, service map, exemplars | | 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards | | 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles | | 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts | @@ -43,10 +43,12 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**. `phase-2.md` for what carries forward. - **3 — Distributed tracing.** The heart of the task: auto- and manual-instrument all seven services, and hand-propagate trace context - across the Redis Stream hop, which carries none by default. + across the Redis Stream hop, which carries none by default. **Done** — see + `phase-3.md`. - **4 — Structured logs, correlated to traces.** Replace prose logging with structured events carrying a lowercase `trace_id`, so Loki and Tempo link - in both directions. + in both directions. **Done** — see `phase-4.md` (OTLP-logs experiment and the + LogQL test migration skipped). - **5 — Metrics.** Instrument RED metrics per service, understand cardinality limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics (404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the