Mark Phases 3 and 4 done, and start Phase 5

Phase 4 records what was built, the two skipped Build steps (OTLP-logs
experiment, LogQL test migration) and what carries forward. Phase 3
records its outcome, with the chaos-nightmare and blocking_event_loop
criteria attested by the user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-13 21:32:43 +03:30
co-authored by Claude Opus 5.5
parent 2c18962243
commit 44d7801617
4 changed files with 60 additions and 15 deletions
+14 -4
View File
@@ -2,11 +2,21 @@
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
**Status:** In progress
**Status:** Done
**Message:** Started 2026-10-12. Phase 2 got `sanctions` tracing through zero-code
`opentelemetry-instrument`, so no `common/telemetry.py` exists yet. Build step 1
writes that bootstrap by hand, in the code, to replace the wrapper.
**Message:** Started 2026-10-12, done 2026-10-13.
- `common/telemetry.py` bootstraps tracing in code for all seven services,
replacing Phase 2's zero-code wrapper. It covers FastAPI, SQLAlchemy, httpx
and Redis instrumentation.
- Trace context crosses the Redis Stream by hand: it's injected into the message
inside a PRODUCER span and extracted by both consumer groups.
- Business spans were added, swallowed exceptions are recorded on their spans,
and the response's `request_id` is now the trace ID. The guard suite is
retired.
- One request produces one connected trace across all seven services, both
consumers included (verified live during Phase 4: 51 spans, 7 services).
- The chaos-nightmare and "explain `blocking_event_loop`" criteria were attested
by the user.
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
+37 -3
View File
@@ -2,10 +2,44 @@
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
**Status:** Not started
**Status:** Done
**Message:** Depends on Phase 3 (trace IDs must exist before log correlation
means anything).
**Message:** Done 2026-10-13. Both directions work in two clicks: a Loki line's
`TraceID` derived field opens the trace, and a Tempo span's logs link returns that
request's lines from every service that logged, the audit/notifier consumers
included (verified live on trace `a4f034b1d66a69b357aef66a647afcef`).
- `common/logging_config.py`: structlog → one JSON object per line on stdout, with
stdlib and uvicorn records going through the same processor chain. Processors
add level, logger, ISO-8601 UTC timestamp, `service`, lowercase `trace_id`/`span_id`
(only inside a valid span) and field-name redaction. `uvicorn.access` is left
alone on purpose, because re-enabling its propagation undoes `--no-access-log`.
- Every call site is now a past-tense event with shared field names (`account_id`,
`transaction_id`, `amount_minor`, `risk_score`, `status_code`, `consumer_group`,
`message_id`, `chaos_flag`). New events: `transaction_failed` (the gateway's 502
path used to log nothing), `transaction_scored`, `holder_screened`,
`stream_message_processed`. `holder_name` is no longer logged at all.
- The czy agent sets `service_name` from the Compose service name. Before that
change, Loki derived it from `container` (`ledgerline-sanctions-1`), so Grafana's
generated `{service_name="sanctions"}` matched nothing. Streams written before
2026-10-13 17:52 UTC keep the old value and don't link.
- **Skipped:** Build step 5 (the OTLP-logs silent-drop experiment; Alloy still has no
`logs` output) and the second Build step 6 (migrating `count_log_lines` to LogQL). The tests run
against the local stack, whose logs never reach Loki, so the migration first
needs a local log path into Loki.
- **Carry forward:**
- The pathology tests still count `docker compose logs` lines matching
`(?i)lookup` (sanctions) and `(?i)healthz` (downstreams). Those words must stay
in exactly `lookup_received` and `healthz_probe_received`.
- `chaos_engaged` lines stay at INFO by choice, so they're visible in Loki during
Phase 9.
- httpx's `HTTP Request:` INFO lines are kept; most of gateway's volume is the
frontend health poll.
- The deploy hot-reloads bind-mounted code into the old image before
`--build`, so a commit that adds a dependency crash-loops the services for
about a minute.
- `alloy-agent`'s config is a single-file bind mount: after `git pull` it needs
`docker compose ... restart alloy-agent` to see the new file.
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
+2 -3
View File
@@ -2,10 +2,9 @@
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
**Status:** Not started
**Status:** In progress
**Message:** Not blocked technically (metrics don't strictly need tracing
first), but sequenced after Phase 4 per the roadmap's ordering.
**Message:** Started 2026-10-13.
**Time-box:** 2 evenings.
+7 -5
View File
@@ -18,9 +18,9 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
| 1 | [`phase-1.md`](phase-1.md) | Done | Deployment: Compose + Traefik on a server | 1 evening | `https://ledgerline.vione.cloud` |
| 2 | [`phase-2.md`](phase-2.md) | Done | Concepts + first trace | 1 evening | One trace in Tempo |
| 3 | [`phase-3.md`](phase-3.md) | In progress | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
| 3 | [`phase-3.md`](phase-3.md) | Done | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 4 | [`phase-4.md`](phase-4.md) | Done | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 5 | [`phase-5.md`](phase-5.md) | In progress | Metrics | 2 evenings | RED per service, service map, exemplars |
| 6 | [`phase-6.md`](phase-6.md) | Not started | Dashboards as code | 2 evenings | `kubectl apply -k k8s/` restores all dashboards |
| 7 | [`phase-7.md`](phase-7.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
| 8 | [`phase-8.md`](phase-8.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
@@ -43,10 +43,12 @@ warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
`phase-2.md` for what carries forward.
- **3 — Distributed tracing.** The heart of the task: auto- and
manual-instrument all seven services, and hand-propagate trace context
across the Redis Stream hop, which carries none by default.
across the Redis Stream hop, which carries none by default. **Done** — see
`phase-3.md`.
- **4 — Structured logs, correlated to traces.** Replace prose logging with
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
in both directions.
in both directions. **Done** — see `phase-4.md` (OTLP-logs experiment and the
LogQL test migration skipped).
- **5 — Metrics.** Instrument RED metrics per service, understand cardinality
limits, and route app metrics to Prometheus given Alloy rejects OTLP metrics
(404 on `/v1/metrics`). Tempo's metrics-generator already produces span metrics and the