mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add deployment phase and Traefik compose override

Browse Source 
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked
Done) between the base project and the OTel concepts phase, and shift
the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md
references (spoiler rules now target Phase 9, the bottleneck hunt).

Add docker-compose.traefik.yml: an override that fronts the gateway with
Traefik and Let's Encrypt and removes all other published host ports.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
2026-10-09 15:38:29 +03:30
co-authored by Claude Sonnet 5.5
parent b2112e55e1
commit 5e514ecf25
17 changed files with 1080 additions and 937 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/roadmap/phase-4.md
+82 -109
View File
@@ -1,126 +1,99 @@
# Phase 4 — Metrics
# Phase 4 — Structured logs, correlated to traces
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
**Status:** Not started
**Message:** Not blocked technically (metrics don't strictly need tracing
first), but sequenced after Phase 3 per the roadmap's ordering.
**Message:** Depends on Phase 3 (trace IDs must exist before log correlation
means anything).
**Time-box:** 2 evenings.
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
### Concepts
- **Instrument types.**
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
connection-pool checkouts.
- **Histogram** — a distribution. Request latency, payload size. This is what
gives you percentiles.
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
read via a callback on collection (right for things you *poll*, like pool size).
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
nonsense. Use histograms.
- **Cardinality — the one thing that will bite you.** Every unique combination of
metric name + label values is a separate time series in memory. Add `account_id`
as a label on a 800-account demo and you've made 800 series; do it in production
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
~10 000 distinct values in production, it does not belong on a metric.** Put it on
a span. Histograms multiply this (one series per bucket per label combination), so
don't over-bucket either.
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
the concrete path), `http.status_code` class, `kind`, `currency`.
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
raw URL, timestamp, error message.
- **RED and USE.**
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
per-service dashboard.
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
connection pools, queues, disk. Saturation is the one everyone forgets and the
one that predicts collapse.
- You want both: RED tells you the service is slow, USE tells you which resource
is why.
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
and knowing both is the point:
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
`servicegraph` connectors read spans in flight and emit RED metrics and edge
metrics. This is what most blog posts (including the OneUptime article) show.
- *In the trace store:* Tempo's **metrics-generator** does the same job after
ingestion, remote-writing the result to Prometheus. **This is what your platform
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
`send_exemplars: true`, remote-writing to a Prometheus started with
`--web.enable-remote-write-receiver`.
- So RED metrics, the service map and exemplars **already exist** before you write
a single metric line. Do not add the collector connectors as well — you'd
duplicate every series under a second set of names. **Go find the real metric
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
they will not match the article's.
- The trade-off worth being able to argue: collector-side generation is cheaper
and works before storage, but sees only what passes through that collector;
Tempo-side generation sees everything that landed, survives collector restarts,
and can be enabled per-tenant — at the cost of load on the trace store.
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
click a spike on a latency graph and land in the exact trace. On your platform
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
`name: trace_id`. If you emit your own histograms with exemplars, the label must
be `trace_id` to match.
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
the pull model is why Prometheus has service discovery and why `up` exists.
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
events: a stable event name plus typed fields (`event="transaction_settled"`,
`account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
Consistent field names across all seven services — `account_id` everywhere, never
`accountId` in one place and `acct` in another. This is what makes LogQL queries
possible at all.
- **Trace–log correlation.** Every log line emitted inside an active span must carry
`trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
"one pane of glass": click a span → see its logs; click a log line → open its
trace. Both directions are configured, and you'll set both up.
- **Two ways to ship logs, and which one your platform actually does:**
- **stdout → collector scraping → Loki.** The container-native way. Your app just
prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
standard in Kubernetes. **This is the only path wired in your Alloy config**
(`loki.source.docker` reading the Docker socket).
- **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
with the same Resource attributes attached automatically. Cleaner correlation,
but the app owns delivery. **On your platform this currently goes nowhere** —
Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
dropped in silence.
- Do both anyway: use stdout for most services, and *extend Alloy* to wire the
OTLP logs path for one or two. Then argue the trade-off from experience. That
comparison — including the fact that a misconfigured pipeline fails silently
rather than loudly — is precisely what Babak will probe.
- **The field name is not a style choice here.** Your Loki datasource is provisioned
with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
A log line must carry lowercase `trace_id`. OpenTelemetry's logging
auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
logs and traces both arriving, no errors anywhere, and no clickable link. Hence
structlog with an explicit processor rather than the auto-injection shortcut.
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
your entire fleet, and Loki bills/indexes on volume.
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
`json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
redaction into the processor chain, not into each call site.
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
not log content. Labels should be few and low-cardinality (`service`, `env`,
`level`). Everything else lives in the log line body and is filtered at query time.
Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
in the JSON body and use a **derived field** to make it clickable.
### Resources
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
Read the first half. Best cardinality explanation available.
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
not a fact.
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
`servicegraph` connector config and its RED PromQL queries almost verbatim.
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
setup, including the hex formatting for trace/span IDs.
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
`OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
fields, if you'd rather not hand-roll a processor.
- Udemy backbone: the Loki + LogQL sections.
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
### Build
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
2. Custom application metrics, deliberately few:
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
- `ledgerline.fraud.score` (Histogram; no labels)
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
metrics, so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add them to
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
playbook. Prometheus scrapes your services directly. Note the irony and the
lesson — you're back to a pull model and a second client library, which is
exactly the fragmentation OTel exists to fix.
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
Alloy so app metrics flow through the collector as the architecture drawing
intends. This also closes a real diagram-vs-code gap in the repo.
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
all three signals.
Do the first to get moving; do one of the others afterwards and write up which
you'd choose for a real platform team and why. That write-up is a Phase 9
artefact.
4. Deliberately break it: add `account_id` as a label to
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
you'll never forget.
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
one. Understand why they differ slightly and when you'd want each.
6. **Second migration of the call-counting assertions.** The three tests you moved
to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
TraceQL query for `sanctions` client spans within a single trace answers
"how many lookups did this one transaction make?" directly, which is the
question the test was always really asking. Do it and compare all three
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
clearest possible demonstration of what each signal is actually for, on a
problem you already understand.
1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
processor that pulls the current span context and injects `trace_id` / `span_id`.
2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
handler: request received (or rely on the span), business decision, error.
3. Add a redaction processor with a denylist of field names.
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
with a `container` label (Alloy's relabel rule sets it from the Docker container
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
the receiver shows traffic, and the pipeline shows no logs output. Now extend
`config.alloy.j2` — add a `logs` output on the receiver plus an
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
That is the most transferable half-hour in the whole project.
6. Confirm the provisioned correlation works in both directions: Loki's
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
are verifying, not writing. If a link doesn't appear, the regex didn't match
your field name — check that before anything else.
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
(`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
in `docker compose logs` — the only cross-service call counter that existed at
Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
with LogQL over structured fields instead. This is a small change and a
pointed one: notice how much shorter and more precise the assertion gets, and
that it now works without shelling out to Docker at all. That delta is the
value of structured logging, measured on your own code.
**Done when:** you have a RED view per service, a working service map, and you can
click a latency spike through to a trace via an exemplar.
**Done when:** from a slow trace you can click into that request's logs, and from
any log line you can click into its trace — in both directions, in two clicks.