mohsen_hassani/LedgerLine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add project documentation and Claude instructions

Browse Source 
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.
This commit is contained in:
Movses Aslanyan
2026-09-05 08:11:00 +03:30
commit eb99cdbb8c
19 changed files with 3372 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PATHOLOGIES.md
+156
View File
@@ -0,0 +1,156 @@
# Pathology catalogue
Thirteen deliberate pathologies, toggled at runtime via the `chaos:flags` Redis
hash (never a restart). Each lives in a `chaos.py` module inside its owning
service. All flags default off.
> Spoiler note (see root `CLAUDE.md`): this file is a complete answer key for
> the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you,
> until that phase is done.
---
## slow_sanctions (`sanctions`)
**Mechanism:** `services/sanctions/chaos.py`. On every `/internal/lookup`
call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of
calls instead sleep a flat 5s -- longer than the 3s downstream timeout every
caller uses (`common/http.py`), so these surface as timeouts one hop up, in
`fraud`.
**Symptom:** heavy-tailed latency on `POST /api/transactions`; most requests
look normal, a fat p95/p99 tail; occasional 504s from `fraud`/`gateway` with
no corresponding 5xx logged in `sanctions` itself (it never errors -- it's
just slow).
## sanctions_errors (`sanctions`)
**Mechanism:** `services/sanctions/chaos.py`. 3% of `/internal/lookup` calls
return HTTP 503 outright, chosen independently per call via `random.random()`.
**Symptom:** a small, steady rate of 502/504 responses from `gateway`, fully
uncorrelated with load or account. `sanctions` logs a "lookup" line for every
attempt regardless, so its request count won't match its success count.
## n_plus_one (`insights`)
**Mechanism:** `services/insights/analytics.py::_n_plus_one_summary`. Instead
of one `LEFT JOIN` against `merchant_categories`, fetches up to 500 raw
transaction rows and issues one additional `SELECT ... FROM merchant_categories
WHERE id = :id` per row.
**Symptom:** `GET /api/accounts/{id}/insights` for a high-volume (hot) account
issues 50-500+ SQL statements instead of 1-2, and its latency grows with the
account's transaction count rather than staying flat.
## pool_starvation (`ledger`)
**Mechanism:** `services/ledger/db.py`. Two engines are built at startup: a
normal one (`pool_size=20, max_overflow=10`) and a starved one (`pool_size=2,
max_overflow=0, pool_timeout=10`). Every `ledger` request picks which
already-running pool to borrow a connection from based on the current flag --
this is what makes the toggle live without a restart, since SQLAlchemy pools
can't be resized in place.
**Symptom:** throughput plateaus as concurrency increases past ~2 in-flight
requests; latency and queueing rise sharply under concurrent load while a
single request in isolation still looks fine.
## row_lock_contention (`ledger`)
**Mechanism:** `services/ledger/repository.py::settle_transaction`. Only for
accounts with `is_hot = true`: takes `SELECT balance_minor FROM accounts ...
FOR UPDATE` and holds the transaction open for an extra 150ms before
committing the balance update.
**Symptom:** concurrent transactions against a hot account show p95 latency
several times higher than the same concurrency spread across many cold
accounts; cold-account traffic is unaffected.
## blocking_event_loop (`fraud`)
**Mechanism:** `services/fraud/chaos.py::synchronous_cpu_work`. Runs ~120ms of
synchronous SHA-256 hashing directly inside the `async def` handler for
`/internal/score` -- no `run_in_executor`/`to_thread`.
**Symptom:** concurrent requests to `fraud` (and anything waiting on it)
serialise: wall-clock time for N concurrent requests approaches N × single-request
time instead of staying close to single-request time, even though nothing is
CPU-bound at the infrastructure level.
## retry_storm (`gateway`)
**Mechanism:** `services/gateway/orchestrator.py::_score_with_retry`. On any
non-2xx from `fraud`, retries immediately up to 3 more times -- no backoff,
no jitter, no circuit breaker.
**Symptom:** when `sanctions` is also failing or slow, the number of
`sanctions` "lookup" log lines per transaction multiplies (up to 4x), and
downstream load spikes exactly when the downstream is already unhealthy --
the opposite of what you want.
## no_batching (`gateway`)
**Mechanism:** `services/gateway/orchestrator.py::create_transaction`. Splits
the transaction into 1-5 synthetic "line items" and calls `fraud` once per
item, sequentially, instead of once for the whole transaction.
**Symptom:** `sanctions` lookups per transaction multiply (roughly 2-5x) and
end-to-end latency for `POST /api/transactions` grows proportionally, even
though sanctions/fraud themselves are healthy.
## unbounded_cache (`insights`)
**Mechanism:** `services/insights/chaos.py` / `app.py`. Instead of the normal
Redis cache (keyed by account + current chaos-flag fingerprint, TTL below),
memoizes summaries in a module-level dict keyed by `(account_id, fingerprint,
time.time_ns())` -- a key that is different on every single call. It never
hits, and it never evicts.
**Symptom:** `insights`' resident memory grows monotonically under repeated
identical-looking requests and never plateaus or drops; cache hit rate (if you
were watching Redis) would be zero -- the "cache" provides no speedup, only
a leak.
## slow_consumer (`notifier`)
**Mechanism:** `services/notifier/consumer.py`. Sleeps 250ms per stream
message before acking, inside the `notifier` consumer-group loop.
**Symptom:** the `notifier` consumer-group lag on `stream:transactions` grows
under sustained load and drains slowly afterward, while the `audit` group
(same stream, independent group) keeps up fine -- isolating the problem to
one specific consumer, not the stream or producer.
## cold_cache (`insights`)
**Mechanism:** `services/insights/app.py`. The Redis summary cache TTL drops
from 900s to 60s.
**Symptom:** repeated identical requests for the same account show periodic
latency spikes roughly every 60s (a full recompute) instead of one initial
spike followed by uniformly fast cache hits.
## poison_data (`ledger`)
**Mechanism:** `services/ledger/app.py` / `repository.py::find_null_currency_row`.
Every `POST /internal/transactions` checks whether the account has any
existing transaction with a `NULL` currency (one of the seeded malformed row
varieties, §8). When the flag is on, that row's `currency` is accessed with no
null guard (`.upper()` on `None`), which raises and surfaces as an unhandled
HTTP 500. When off, the same check is guarded and does nothing.
**Symptom:** a specific, small set of accounts reliably 500 on every
transaction attempt while the flag is on, and behave completely normally
while it's off -- input data that was always there, suddenly fatal.
## chatty_health (`gateway`)
**Mechanism:** `services/gateway/orchestrator.py::_run_chatty_health_probe`.
Before doing any real work, `POST /api/transactions` calls `GET /healthz`
sequentially on all five downstreams, tagged with an `X-Ledgerline-Probe`
header so the extra hits are distinguishable from routine health polling.
**Symptom:** every transaction request produces five extra, sequential
health-check round trips before the real work even starts, inflating latency
by a fixed per-request tax that has nothing to do with load or data size.
docs/phase-0-base-project-spec.md
+373
View File
@@ -0,0 +1,373 @@
# Phase 0 — Technical Spec: "Ledgerline" base project
> **How to use this file:** paste the whole thing into your coding agent as the task
> spec. It is written to be executed without further clarification. Read
> §2 (Non-goals) yourself before you start — the deliberate omissions are the
> point of the exercise.
---
## 1. Mission
Build **Ledgerline**, a deliberately imperfect microservice system that simulates a
small payments/finance backend. It exists for one reason: to be a realistic subject
for observability work in later phases. The business logic is fake and does not
need to be correct, defensible, or secure. What it *does* need to be is
**structurally realistic** — real network hops, real database queries, real async
message passing, real contention, real latency distributions, and real failure modes.
The system must be runnable with `make up && make seed && make smoke` on a laptop.
---
## 2. Non-goals — read carefully
These are **hard constraints**, not preferences. Violating them destroys the value
of the project.
- **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client,
no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no
APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
- **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add
indexes, connection-pool tuning, batching, backoff, or caching beyond what is
specified.
- **DO NOT add authentication, authorization, rate limiting, or input hardening.**
A single hardcoded demo user is fine.
- **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the
deliverable.
- **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only.
- **DO NOT abstract prematurely.** Duplication across services is acceptable and
preferred over a clever shared framework.
The only logging permitted is Python's standard `logging` module with a plain-text
formatter (§9).
---
## 3. Technology
| Concern | Choice |
|---|---|
| Language | Python 3.12 |
| Web framework | FastAPI + `uvicorn` (one worker per service) |
| HTTP client | `httpx` (async) |
| DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` |
| Cache / queue | Redis 7 (`redis-py` asyncio client) |
| Async messaging | Redis **Streams** (not Celery, not RabbitMQ) |
| Validation | Pydantic v2 |
| Packaging | One shared `Dockerfile`, one image, `CMD` differs per service |
| Orchestration | Docker Compose |
| Schema | A single idempotent `db/schema.sql` — no Alembic |
| Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework |
Redis Streams is chosen deliberately: it has **no automatic context propagation**,
so a later phase has to implement it manually. Do not swap it for something with
built-in instrumentation.
---
## 4. Services
Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`.
| # | Service | Port | Role | Stores |
|---|---|---|---|---|
| 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) |
| 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
| 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
| 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) |
| 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis |
| 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres |
| 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |
Every HTTP service exposes:
- `GET /healthz` → `{"status": "ok", "service": "<name>"}` — liveness, no dependency checks
- `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable
- `GET /` → `{"service": "<name>", "version": "0.1.0"}`
---
## 5. Domain model (`db/schema.sql`)
```
accounts (id UUID pk, holder_name text, currency char(3),
balance_minor bigint, is_hot boolean default false,
created_at timestamptz)
merchant_categories (id smallint pk, code text unique, label text,
risk_weight numeric(3,2))
transactions (id UUID pk, account_id UUID, kind text, -- deposit|withdrawal|purchase
amount_minor bigint, currency char(3),
merchant_name text, category_id smallint,
status text, -- pending|settled|rejected
risk_score numeric(4,3), created_at timestamptz)
sanctions_entries (id serial pk, normalized_name text, country char(2),
list_source text, added_at timestamptz)
notifications (id UUID pk, transaction_id UUID, channel text, -- email|push|sms
status text, attempts int, sent_at timestamptz)
audit_events (id bigserial pk, transaction_id UUID, event_type text,
payload jsonb, recorded_at timestamptz)
```
**Index policy — important.** Create only these indexes:
- primary keys (implicit)
- `accounts(holder_name)`
- `sanctions_entries(normalized_name)`
Deliberately **omit** `transactions(account_id, created_at)` and
`transactions(category_id)`. Put the `CREATE INDEX` statements for those in
`db/optional_indexes.sql`, applied only by `make fix-index`.
---
## 6. Request flows
### 6.1 `POST /api/transactions` — the primary flow
The trace this produces should be ~4 levels deep with one async fork.
```
browser
└─ gateway POST /api/transactions
├─ ledger POST /internal/transactions (create as `pending`, row lock on account)
├─ fraud POST /internal/score (risk scoring)
│ └─ sanctions POST /internal/lookup (watchlist check — slow vendor)
├─ ledger POST /internal/transactions/{id}/settle (or /reject if risk_score > 0.8)
├─ redis XADD stream:transactions (fire-and-forget)
│ ├─ notifier consumer group `notifier`
│ └─ audit consumer group `audit`
└─ insights GET /internal/accounts/{id}/summary (cached)
```
Response body:
```json
{
"transaction_id": "…", "status": "settled", "risk_score": 0.12,
"balance_minor": 481200, "summary": { "…": "…" },
"request_id": "…", "elapsed_ms": 843
}
```
### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow
`gateway → insights`. `insights` runs a 120-day aggregation over `transactions`.
This is where the N+1 and the missing index bite.
### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown
### 6.4 Admin
- `GET /api/admin/chaos` → current flag values
- `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}`
- `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}`
Flags live in a Redis hash `chaos:flags` so every service reads the same state
without a restart. Each service reads the hash on **every request** (a tiny bit of
extra Redis load is fine and realistic).
---
## 7. The pathology catalogue
Each pathology lives in a clearly named `chaos.py` module inside its owning service,
with a docstring naming it. All flags default **off**.
| Flag | Service | Behaviour when enabled |
|---|---|---|
| `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). |
| `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. |
| `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. |
| `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) |
| `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. |
| `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. |
| `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. |
| `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. |
| `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. |
| `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. |
| `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. |
| `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. |
| `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. |
**Profiles**
- `clean` — all off.
- `light` — `slow_sanctions`, `cold_cache`.
- `nightmare` — all thirteen on.
- `random` — pick a random subset of size 3–6, using a seed derived from
`time.time()` and **not logged anywhere**. Store the active set in Redis under
`chaos:flags` only, never write it to a file or print it. (This makes the later
bottleneck-hunting exercise a genuine hunt.)
Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the
symptom a reader should expect. Do **not** reproduce or summarise its contents in
your final chat response.
---
## 8. Seed data (`scripts/seed.py`)
Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert.
- **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD,
10% GBP. Balances log-normal, median ≈ 250 000 minor units.
Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets.
- **45 merchant categories** with `risk_weight` between 0.01 and 0.95.
- **250 000 transactions** spread over the last 120 days.
- Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
- Amounts log-normal.
- Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest
on a Zipf-ish distribution.
- `status` 92% `settled`, 5% `rejected`, 3% `pending`.
- **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account
holders, so a meaningful fraction of requests genuinely score as high risk.
- **~2% malformed rows** distributed through `transactions`:
- `currency` = `NULL`
- negative `amount_minor` on a `deposit`
- `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence
- `created_at` set 3 days in the future
- `category_id` pointing at a non-existent category
Print a summary table at the end (row counts per table, hot account IDs, count of
malformed rows). Seeding 250k rows must complete in under 60 seconds — use
`COPY` / `executemany`, not per-row inserts.
---
## 9. Logging (the deliberately bad baseline)
Every service configures:
```python
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
```
- Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value.
- `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it
as a header on every downstream `httpx` call. Each service logs it manually at the
start of each handler. **This is the only correlation mechanism that exists.**
It is intentionally weak — it does not survive the Redis Stream hop, it carries no
timing, and it cannot express parent/child relationships.
- Set `PYTHONUNBUFFERED=1` in the Dockerfile.
- Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK.
---
## 10. Frontend
Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`,
`app.js`, `style.css`. No bundler, no npm, no CDN dependencies.
Layout, top to bottom:
1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`),
amount input, merchant name input, kind radio (deposit / withdrawal / purchase),
Submit button.
2. **Result panel** — pretty-printed JSON response, plus the browser-measured
round-trip time in ms, plus the `request_id` shown in a monospace font that is
easy to copy.
3. **Load burst** — a button that fires *N* concurrent requests (N selectable:
10 / 50 / 200) against random accounts, then renders a small table:
count, success, errors, min / mean / p50 / p95 / max ms, measured client-side.
This is a crude stand-in for a real load tool; a later phase replaces it with k6.
4. **Chaos panel** — a checkbox per flag reflecting current state from
`GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile
buttons (`clean`, `light`, `nightmare`, `random`).
5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s
via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green
or red dot per service.
Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.
---
## 11. Repository layout
```
ledgerline/
├─ Makefile
├─ docker-compose.yml
├─ Dockerfile # single image for all Python services
├─ pyproject.toml # or requirements.txt — one dependency set
├─ .env.example
├─ README.md
├─ db/
│ ├─ schema.sql
│ └─ optional_indexes.sql
├─ scripts/
│ ├─ seed.py
│ └─ smoke.py
├─ common/
│ ├─ __init__.py
│ ├─ settings.py # pydantic-settings, env-driven
│ ├─ db.py # async engine/session factory
│ ├─ redis_client.py
│ ├─ http.py # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
│ └─ logging_config.py # the plain-text logging setup from §9
├─ services/
│ ├─ gateway/ (app.py, chaos.py, orchestrator.py, static/)
│ ├─ ledger/ (app.py, chaos.py, repository.py)
│ ├─ fraud/ (app.py, chaos.py, scoring.py)
│ ├─ sanctions/ (app.py, chaos.py, repository.py)
│ ├─ insights/ (app.py, chaos.py, analytics.py)
│ ├─ notifier/ (app.py, chaos.py, consumer.py)
│ └─ audit/ (worker.py, consumer.py)
└─ docs/
└─ PATHOLOGIES.md
```
`docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use
`depends_on` with `condition: service_healthy` for postgres and redis. Mount the
source read-write and run uvicorn with `--reload` so iteration is fast.
Reserve — but leave empty and unreferenced — a top-level `observability/` directory
with a `.gitkeep`. Later phases fill it.
---
## 12. Makefile targets
| Target | Effect |
|---|---|
| `make up` / `make down` | compose up -d / down |
| `make logs` | `docker compose logs -f` |
| `make seed` | apply `db/schema.sql`, run `scripts/seed.py` |
| `make reset` | `down -v`, `up`, `seed` |
| `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s |
| `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile |
| `make fix-index` | apply `db/optional_indexes.sql` |
| `make drop-index` | drop them again |
---
## 13. Acceptance criteria
The task is done when all of these hold on a clean checkout:
1. `make reset` completes in under 3 minutes and prints the seed summary table.
2. `make smoke` passes with `chaos-clean`.
3. The frontend loads at `http://localhost:8000`, the account dropdown is populated,
and a submitted transaction returns a settled or rejected result.
4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under
~250 ms and zero errors.
5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error
rate between 2% and 10% — degraded but not fully broken.
6. Toggling any single flag via the chaos panel changes behaviour **without a
restart**.
7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`.
8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns
nothing.
9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies.
In your final message, report only: services started, row counts seeded, the
p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you
deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.
docs/phase-0a-test-suite-spec.md
+508
View File
@@ -0,0 +1,508 @@
# Phase 0a — Technical Spec: Ledgerline integration test suite
> **How to use this file:** paste it into the **test-authoring agent** together
> with `phase-0-base-project-spec.md`. That agent writes only `tests/`.
> A second agent then implements the services against the resulting suite.
> See §1 for the workflow contract between them.
---
## 1. The two-agent workflow
### Agent A — test author (this spec)
- Reads `phase-0-base-project-spec.md` (the system contract) and this file.
- Creates **only** these paths:
```
tests/** pytest.ini (or pyproject [tool.pytest])
requirements-test.txt
tests/README.md
tests/RED-BASELINE.md
```
Plus the `test-*` targets appended to the `Makefile`.
- Creates **nothing** under `services/`, `common/`, `db/`, `scripts/`,
`observability/` or `loadtest/`. Not even a stub. If a test needs a fixture
from the system, it asserts against the contract in §4 rather than importing
application code.
- **Imports nothing from the application.** Tests talk to the system over HTTP,
Postgres, Redis, and the Docker CLI. No `from services.gateway import …`.
This is what keeps them behavioural rather than implementation-coupled.
- Runs the suite against an empty repo and confirms it is **red**, then writes
`tests/RED-BASELINE.md` listing, per test, *why* it currently fails
(connection refused / 404 / assertion / missing table). A test that passes
against nothing is a broken test — find it and fix it.
### Agent B — implementer
- Reads `phase-0-base-project-spec.md` and treats `tests/` as a **read-only
contract**.
- **Must not modify, delete, skip, xfail, or loosen any test**, or change
`pytest.ini`, markers, thresholds, or fixtures. If a test appears wrong,
impossible, or contradictory: **stop and report it**, don't edit it.
- Must implement each pathology by the mechanism described in
`phase-0-base-project-spec.md` §7 — not as a generic `sleep()` tuned to make a
timing assertion pass. Several assertions in §6 are deliberately built so that
faking is harder than implementing (query counts, `EXPLAIN` plan shapes, Redis
consumer lag, process RSS). Treat that as the intent everywhere, including
where the assertion happens to be simpler.
- Done when `make test` is green three consecutive runs.
---
## 2. Design principles for this suite
These are the rules the tests must obey. They matter more than any individual
test.
1. **Assert goals, not implementation.** No test may reference a function name,
module path, class, SQL string, log format, or internal attribute. A test may
reference the *public contract*: HTTP routes, status codes, JSON field names,
table names, Redis stream and consumer-group names, chaos flag names.
2. **Prefer assertions that are hard to satisfy without the real mechanism.**
Wall-clock latency alone is fakeable with `sleep()`. Query counts from
`pg_stat_statements`, `EXPLAIN` plan node types, Redis consumer-group lag,
container RSS, and log-line counts on a *different service* are not. Where
both are available, assert on both.
3. **Measure differentially, never absolutely.** With one exception (§5.7),
every pathology test compares flag-on against flag-off in the same run, on
the same machine, and asserts a *ratio*. Absolute millisecond thresholds do
not survive a different laptop.
4. **Design for a noisy laptop.** Generous margins, warm-up requests discarded,
median-of-N measurement runs. A test that fails when Slack is open is worse
than no test.
5. **Fast tests and slow tests are separate.** `make test-contract` must run in
under 60 seconds. Pathology tests may take minutes and live behind their own
marker.
6. **Every test restores state.** A test that enables a chaos flag must reset
the profile in teardown, even on failure.
---
## 3. Stack, layout, and execution
Tests run **on the host**, against the Compose stack on `localhost`. They do not
run inside a container and do not use testcontainers — the stack is started by
`make up` beforehand.
```
tests/
├─ README.md
├─ RED-BASELINE.md
├─ conftest.py # fixtures: clients, chaos control, seed manifest
├─ harness/
│ ├─ load.py # concurrent request driver + LatencyReport
│ ├─ pg.py # pg_stat_statements deltas, EXPLAIN helpers
│ ├─ streams.py # Redis XINFO GROUPS helpers
│ └─ containers.py # docker stats / docker logs helpers
├─ test_contract_health.py
├─ test_contract_transactions.py
├─ test_contract_insights.py
├─ test_contract_async.py
├─ test_contract_chaos.py
├─ test_contract_frontend.py
├─ test_seed_data.py
├─ test_baseline_health.py
├─ test_pathologies.py
├─ test_nightmare.py
└─ test_no_observability.py
```
Dependencies (`requirements-test.txt`): `pytest`, `pytest-asyncio`,
`pytest-timeout`, `httpx`, `psycopg[binary]`, `redis`. Docker access via
`subprocess` calling the `docker` CLI — no SDK.
### Markers
| Marker | Runtime | Contents |
|---|---|---|
| `contract` | < 60 s | Functional behaviour, chaos-clean |
| `seed` | < 20 s | Seed data integrity |
| `baseline` | < 90 s | The clean system is genuinely healthy |
| `pathology` | 5–12 min | One test per pathology, differential |
| `slow` | — | Additional marker on the soak-ish ones (`unbounded_cache`, `cold_cache`) |
| `guard` | < 5 s | The "no observability yet" assertions |
Configure `pytest.ini` with `--strict-markers`, `asyncio_mode = auto`, and a
global `timeout = 300`.
### Make targets to append
| Target | Command |
|---|---|
| `test` | everything except `slow` |
| `test-contract` | `-m "contract or seed or guard"` |
| `test-baseline` | `-m baseline` |
| `test-pathology` | `-m "pathology and not slow"` |
| `test-slow` | `-m slow` |
| `test-all` | no marker filter |
---
## 4. Contract additions required for testability
These extend `phase-0-base-project-spec.md`. Agent B must implement them; Agent A
may assume them.
### 4.1 Seed manifest
`scripts/seed.py` writes `db/seed_manifest.json` **and** a `seed_manifest` table
(single row, `data jsonb`) with the same content. Tests read the table, so they
don't depend on a host file path.
```json
{
"seed": 42,
"generated_at": "…",
"counts": { "accounts": 800, "transactions": 250000,
"merchant_categories": 45, "sanctions_entries": 12000,
"malformed_transactions": 5013 },
"hot_account_ids": ["…", "…", "…", "…", "…"],
"cold_account_ids_sample": ["…"], // 50 non-hot accounts, for spread tests
"sanctioned_account_ids": ["…"], // accounts whose holder matches the watchlist
"poison_account_ids": ["…"], // accounts owning malformed rows
"clean_account_id": "…" // no malformed rows, not hot, not sanctioned
}
```
### 4.2 Response envelope
`POST /api/transactions` → **201** on success:
```json
{ "transaction_id": "<uuid>", "status": "settled|rejected",
"risk_score": 0.0-1.0, "balance_minor": 0,
"summary": { }, "request_id": "<uuid>", "elapsed_ms": 0 }
```
Errors return a consistent envelope with the appropriate status:
```json
{ "error": "<machine_readable_code>", "detail": "<human string>",
"request_id": "<uuid>" }
```
Status mapping: `400` invalid input, `404` unknown account, `422` validation,
`502` downstream returned an error, `504` downstream timed out, `500`
unhandled — which is exactly what `poison_data` produces, and what §6.12 asserts.
### 4.3 Chaos API
- `GET /api/admin/chaos` → `{"profile": "<name|custom>", "flags": {"<flag>": bool, …}}`
containing **all thirteen** flags, always.
- `POST /api/admin/chaos` → `{"flag": "<name>", "enabled": bool}`; returns the
full new state. Unknown flag → `400`.
- `POST /api/admin/chaos/profile` → `{"profile": "clean|light|nightmare|random"}`;
returns the full new state. For `random`, the response reports the flags —
**but `make chaos-random` must not print them**, and the seed is never logged.
- `GET /api/health/all` → `{"services": {"<name>": "ok"|"unhealthy", …}}` for all
six HTTP services.
### 4.4 `pg_stat_statements`
The Postgres image must load `pg_stat_statements` (`shared_preload_libraries`)
and the extension must be created by `db/schema.sql`. Tests use
`pg_stat_statements_reset()` around a single request to count queries.
### 4.5 Deterministic seeding
`SEED` env var, default `42`. Same seed ⇒ identical manifest, identical row
counts, identical hot/poison account IDs.
---
## 5. Goal-to-test traceability
Every test must map to one of these goals. Put the goal ID in the test's
docstring. This is what keeps the suite about intent rather than mechanics.
| ID | Goal | Where |
|---|---|---|
| **G1** | A transaction traverses all seven services and returns a coherent result | §5.1 |
| **G2** | The system is genuinely multi-service, not a collapsed monolith | §5.2 |
| **G3** | Async work happens after the response, via two independent consumer groups | §5.3 |
| **G4** | The seed data is large, skewed, and contains the intended poison rows | §5.4 |
| **G5** | Chaos flags toggle at runtime without a restart | §5.5 |
| **G6** | Each pathology measurably degrades the system when enabled | §6 |
| **G7** | The clean baseline is actually healthy — otherwise G6 means nothing | §5.6 |
| **G8** | Nightmare mode degrades but does not collapse | §5.7 |
| **G9** | Failure is contained: timeouts fire, the gateway doesn't wedge | §5.8 |
| **G10** | No observability tooling is present yet | §5.9 |
| **G11** | The frontend and its supporting endpoints are served | §5.10 |
### 5.1 G1 — end-to-end transaction (`contract`)
- Submitting a valid transaction for `clean_account_id` returns 201 with the full
envelope; `transaction_id` is a UUID; `status` ∈ {settled, rejected};
`risk_score` ∈ [0,1]; `elapsed_ms` > 0.
- The transaction is durable: a row with that ID exists in `transactions` with a
terminal status within 2 s.
- The balance moves in the correct direction for a deposit and for a withdrawal,
by exactly `amount_minor`, when the transaction settles.
- A transaction for a `sanctioned_account_ids` holder yields a materially higher
`risk_score` than one for `clean_account_id` (assert the ordering, not a value).
- `GET /api/accounts/{id}/insights` returns a summary for a seeded account.
- Unknown account → 404 with the error envelope. Negative amount on a deposit →
400 or 422. Missing required field → 422.
### 5.2 G2 — really seven services (`contract`)
- All six HTTP services answer `/healthz` on their own port with their own name.
- `/readyz` returns 200 for all six.
- `GET /api/health/all` reports all six as ok.
- Stopping `sanctions` (`docker compose stop sanctions`) causes
`GET /api/health/all` to report it unhealthy within 15 s, and
`POST /api/transactions` to fail with 502 or 504 — **not** 200. Restart it in
teardown and wait for ready.
*(This is the test that catches a collapsed monolith: if transactions still
succeed with `sanctions` stopped, the service isn't really in the path.)*
### 5.3 G3 — asynchronous work (`contract`)
- After a successful transaction, a row appears in `notifications` **and** in
`audit_events` for that `transaction_id` within 10 s.
- Both appear — proving two independent consumer groups, not one consumer doing
both jobs.
- `XINFO GROUPS stream:transactions` reports exactly two groups, named
`notifier` and `audit`.
- The API response returns **before** the notification row exists at least once
across 20 attempts, proving the work is genuinely off the request path.
*(Sample the DB immediately on response; assert at least one miss. Tolerant of
a fast machine winning the race sometimes — hence 20 attempts, not 1.)*
### 5.4 G4 — seed data (`seed`)
Against the manifest and the database:
- Row counts match the manifest and the spec's targets within ±2%.
- Exactly 5 accounts have `is_hot = true`, and they match `hot_account_ids`.
- The 5 hot accounts own between 20% and 40% of all transactions.
- Transactions span ≥ 110 distinct days.
- Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each).
- Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points).
- Malformed rows exist in all five described varieties: null currency, negative
amount on a deposit, non-ASCII merchant name, future `created_at`, orphan
`category_id`. Assert **at least one of each**, and 1%–4% overall.
- ≥ 30 sanctions entries match seeded holder names.
- Re-running the seed with the same `SEED` yields an identical manifest
(compare the `counts` and `hot_account_ids`).
- Neither `transactions(account_id, created_at)` nor `transactions(category_id)`
exists in `pg_indexes` after a fresh seed.
### 5.5 G5 — chaos control (`contract`)
- `GET /api/admin/chaos` lists all thirteen flags by name.
- Toggling a single flag changes only that flag.
- Each profile sets the expected shape: `clean` → all false; `nightmare` → all
true; `light` → exactly the two documented flags.
- **No restart:** with `chaos-clean` set, record the `insights` latency; enable
`n_plus_one` via the API; without restarting any container, the query count for
the same request rises. Assert on the query-count delta (§6.3's helper), so
this test proves live reconfiguration rather than timing noise.
- Unknown flag name → 400.
- Teardown restores `clean`.
### 5.6 G7 — the clean baseline is healthy (`baseline`)
With `chaos-clean`, after 20 discarded warm-up requests, drive 200 requests at
concurrency 10 against random cold accounts:
- error rate == 0
- p95 < 400 ms *(the one absolute threshold in the functional set; if it fails on
a slow machine, that machine cannot run the pathology suite meaningfully
either, and the failure message must say so)*
- p99 / p50 < 5 — the clean system has no fat tail
- `insights` for a cold account issues fewer than 10 SQL statements
- Redis consumer lag returns to 0 within 15 s of the burst finishing
### 5.7 G8 — nightmare degrades but survives (`pathology`)
Same driver, `chaos-clean` then `chaos-nightmare`, 300 requests at concurrency 20
each:
- nightmare p95 ≥ **5×** clean p95
- nightmare error rate between **2% and 15%** — degraded, not dead
- at least 80% of requests still return a valid envelope (either success or a
structured error, never a connection reset or an empty body)
- `GET /api/health/all` still responds within 5 s during the burst
### 5.8 G9 — failure containment (`contract`)
- With `sanctions_errors` at its maximum and `retry_storm` off, no request takes
longer than 10 s. The 3 s downstream timeout must actually fire.
- With `sanctions` stopped entirely, `POST /api/transactions` returns 502/504
within 10 s, and the gateway still answers `/healthz` in under 1 s throughout.
- No request ever returns a 200 with a `null` `transaction_id`.
### 5.9 G10 — no observability yet (`guard`)
Static and runtime assertions that the Phase 0 "before" state is intact:
- No file under `services/` or `common/` imports `opentelemetry`,
`prometheus_client`, `structlog`, `sentry_sdk`, or `elasticapm`.
- Neither `requirements*.txt` nor `pyproject.toml` lists them (excluding
`requirements-test.txt`).
- No service exposes `/metrics` (assert 404 on all six).
- `docker compose logs gateway` emits plain text, not JSON: sample 20 lines and
assert none parse as a JSON object.
> This test class is **expected to be deleted at the start of Phase 2**, not
> maintained. Say so in its module docstring.
### 5.10 G11 — frontend (`contract`)
- `GET /` returns 200, `text/html`, and references `app.js` and `style.css`.
- Those two assets return 200 with sensible content types.
- `GET /api/accounts` returns ≥ 100 accounts, each with `id`, `holder_name`,
`currency`, `balance_minor`.
No browser automation. Serving and shape only.
---
## 6. G6 — pathology tests
**One test per flag.** Each follows the same skeleton:
```
1. set chaos-clean
2. warm up (discard)
3. measure → baseline
4. enable exactly one flag via the API
5. warm up (discard)
6. measure → degraded
7. assert the ratio, with the failure message printing both measurements
8. teardown: chaos-clean
```
Measurement runs the median of 3 for latency-based assertions. Failure messages
must always include both raw numbers — a bare `assert ratio > 3` is useless at
2 a.m.
| § | Flag | Primary signal (hard to fake) | Secondary signal | Assertion |
|---|---|---|---|---|
| 6.1 | `slow_sanctions` | — | p99/p50 of `POST /api/transactions` | degraded p99/p50 ≥ **6**, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99 |
| 6.2 | `sanctions_errors` | count of 502/504 responses over 300 requests | — | degraded error rate ∈ [1%, 10%]; baseline == 0 |
| 6.3 | `n_plus_one` | `pg_stat_statements` call delta across **one** insights request | latency ratio | degraded ≥ **50** statements; baseline ≤ 10; latency ratio ≥ 2 |
| 6.4 | *missing index* (`make drop-index` / `fix-index`, not a chaos flag) | `EXPLAIN (ANALYZE, FORMAT JSON)` on the insights aggregation contains a `Seq Scan` on `transactions`; after `fix-index` it contains an `Index Scan`/`Bitmap Index Scan` | latency ratio | plan node types as stated; unindexed latency ≥ **3×** indexed. Teardown restores the dropped state. |
| 6.5 | `pool_starvation` | throughput (req/s) at concurrency 5 vs 25 | p95 ratio | with the flag on, throughput at c=25 is ≤ **1.4×** throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25 |
| 6.6 | `row_lock_contention` | 25 concurrent transactions against **one hot account** vs 25 spread across 25 cold accounts, both with the flag on | — | single-hot-account p95 ≥ **3×** spread p95; with the flag off the same comparison is ≤ 1.8× |
| 6.7 | `blocking_event_loop` | wall-clock for 10 concurrent requests ÷ wall-clock for 1 request | — | degraded ratio ≥ **6** (serialised); baseline ratio ≤ 2.5 (concurrent) |
| 6.8 | `retry_storm` | count of `sanctions` container log lines matching a lookup, per transaction, with `sanctions_errors` also on | — | degraded ≥ **2.0** lookups/transaction; baseline (retry off, errors on) ≤ 1.2. *Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.* |
| 6.9 | `no_batching` | `sanctions` lookup log lines per transaction, `sanctions_errors` **off** | latency ratio | degraded ≥ **2.0** lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5 |
| 6.10 | `unbounded_cache` (`slow`) | `insights` container RSS via `docker stats` before vs after 2000 requests with **distinct** query parameters, sampled 30 s after the burst ends | — | degraded RSS growth ≥ **20 MB** and still elevated at +30 s; baseline growth ≤ 8 MB |
| 6.11 | `slow_consumer` | `XINFO GROUPS stream:transactions` → `lag`/`pending` for the `notifier` group after a 150-transaction burst | drain time | degraded lag ≥ **30** at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s |
| 6.12 | `cold_cache` (`slow`) | same insights request at 1 rps for 150 s; count samples > 3× the median | — | degraded ≥ **2** spikes (a 60 s TTL over 150 s); baseline ≤ 1 |
| 6.13 | `poison_data` | request against each `poison_account_ids` entry | — | degraded produces ≥ 1 response with status **500**; baseline produces zero 500s across the same requests (4xx or 200 only) |
| 6.14 | `chatty_health` | `/healthz` hits on downstream containers during a single transaction, counted from logs | — | degraded ≥ **5** healthz lines across downstreams per transaction; baseline == 0 |
**Note on log-line counting (6.8, 6.9, 6.14).** At Phase 0 the plain-text log is
the only cross-service observability surface that exists — which is precisely the
deficiency this whole project sets out to fix. Implement these helpers behind
`harness/containers.py::count_log_lines(service, pattern, since)` so there is
exactly one place to change. **From Phase 3 these assertions should be rewritten
against structured logs, and from Phase 4 against span counts** — at which point
they become both simpler and more robust. Note this in `tests/README.md`; it is a
deliberate teaching artefact, not a shortcut.
**Anti-fake note for Agent B.** 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied
by inserting a `sleep()`: they require, respectively, real repeated queries, a
real missing index, real unbounded allocation, real consumer slowness, and a real
unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12)
are stated as ratios for the same reason — passing them by faking is more work
than implementing the mechanism described in the Phase 0 spec.
---
## 7. The measurement harness
`harness/load.py` provides one primitive:
```python
async def drive(
method: str, url_factory: Callable[[], tuple[str, dict]],
total: int, concurrency: int,
warmup: int = 20, timeout: float = 15.0,
) -> LatencyReport
```
`LatencyReport` carries `count, ok, errors, error_rate, status_counts,
p50, p95, p99, max, throughput_rps, wall_seconds`, plus a `__str__` suitable for
dropping straight into an assertion message.
Rules:
- Warm-up requests are issued and discarded before timing starts.
- Percentiles from the raw sorted sample, nearest-rank. No interpolation, no
averaging of percentiles across runs — **median of the p95s**, never the mean.
- A closed-loop driver (fixed concurrency) is used deliberately for its
simplicity here. Note in `tests/README.md` that this under-measures latency
under saturation (coordinated omission) and that Phase 6 replaces it with k6's
arrival-rate executors. Don't try to fix it in the test suite.
- `measure_median(n=3, fn)` helper for the pathology tests.
`harness/pg.py`:
```python
@contextmanager
def statement_count(dsn) -> Counter # resets pg_stat_statements, yields, returns delta
def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON)
def plan_node_types(plan: dict) -> set[str]
```
`harness/streams.py`: `group_lag(group)`, `wait_for_drain(group, timeout)`.
`harness/containers.py`: `rss_mb(service)`, `count_log_lines(service, pattern,
since)`, `stop(service)`, `start(service)`, `wait_ready(service, timeout)`.
---
## 8. Fixtures (`conftest.py`)
| Fixture | Scope | Purpose |
|---|---|---|
| `stack_up` | session | Assert the Compose stack is running and all six services are ready; **fail fast with a clear message** ("run `make up && make seed` first") rather than timing out |
| `seeded` | session | Assert the seed manifest exists and counts are non-zero; fail fast otherwise |
| `manifest` | session | Parsed seed manifest |
| `api` | function | `httpx.AsyncClient` against the gateway, 15 s timeout |
| `dsn` | session | Postgres DSN from env |
| `redis_client` | session | — |
| `chaos` | function | `chaos.set_profile(name)`, `chaos.set_flag(name, bool)`, `chaos.get()`. **Autouse teardown resets to `clean`**, even on failure |
| `fresh_balances` | function | Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs |
Every environment value (ports, DSN, Redis URL) comes from env with the Compose
defaults as fallbacks. No hardcoded values buried in test bodies.
---
## 9. `tests/README.md` — what Agent A must write
- How to run each marker set, and the prerequisite (`make up && make seed`).
- The measurement methodology and why ratios are used rather than absolutes.
- The known-flaky list: 6.10 and 6.12 are the least stable; document the retry
strategy and what a genuine failure looks like versus noise.
- The coordinated-omission caveat on the closed-loop driver.
- The Phase 3/4 migration note for the log-counting helpers.
- An explicit statement that `test_no_observability.py` is deleted at Phase 2.
---
## 10. Acceptance criteria
For **Agent A**:
1. `pytest --collect-only` succeeds and every test carries exactly one goal ID in
its docstring and at least one marker.
2. `grep -rn "from services\|from common\|import services\|import common" tests/`
returns nothing.
3. Every §5 and §6 item has a corresponding test; `tests/README.md` contains the
goal-to-test table.
4. The suite is **red** against an empty repo, and `tests/RED-BASELINE.md`
records the failure reason per test. No test passes vacuously.
5. `make test-contract` completes (failing) in under 60 s.
For **Agent B**:
1. `make test` green three consecutive runs.
2. `make test-slow` green (may be run once).
3. `git diff --stat tests/` against Agent A's commit is empty.
4. `docs/PATHOLOGIES.md` written, and not summarised in the final response.
docs/roadmap/intro.md
+377
View File
@@ -0,0 +1,377 @@
# Observability & APM — Delivery Roadmap for Babak's Task
*Part of `docs/roadmap/` — platform background and reading guidance shared by
every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`;
[`summary.md`](summary.md) gives the one-page overview and status of all of
them.*
Ten phases. Each one: concepts first, then the build, then a checkable exit
condition. Total time-boxed at roughly **4–6 weeks part-time** (evenings +
weekends). Nothing here is theory for theory's sake — every concept block is
followed immediately by the place it lands in the code.
---
## Read this first — the four questions blocking you
### "What *is* an APM? Do you install it, use it, implement it, or build it?"
APM (Application Performance Monitoring) is a **product category and a capability**,
not a protocol or a single tool. A system has APM when you can answer, for
production traffic: what's the request rate, error rate and latency distribution
per endpoint; which services call which; where did *this specific* request spend its
time; which database query or external call was slow; and what error was thrown,
with the stack.
The verb depends on the vendor model:
- **Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM).** You
*install* an agent that attaches to your runtime and ships to their backend. Mostly
a deployment exercise. Turnkey, expensive, proprietary.
- **Grafana stack — what Babak asked for.** There is no product called "Grafana APM"
that you install. You **assemble** the capability from four layers:
| Layer | What you do | Component |
|---|---|---|
| Instrumentation | *Implement* — this is code you write | OpenTelemetry SDK inside your services |
| Pipeline | *Deploy + configure* | OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution) |
| Storage | *Deploy* | Tempo (traces), Loki (logs), Prometheus or Mimir (metrics) |
| Query & UI | *Configure* | Grafana + provisioned datasources with cross-links |
So: you **implement** instrumentation, you **deploy** the pipeline and stores, you
**configure** the correlation. The "APM" is the emergent property of all four
working together.
The distinction that actually matters: APM data is generated *inside* the
application, keyed to *individual requests*, and *correlated* across signals.
Infrastructure monitoring (CPU, disk, pod restarts) is generated *outside* the
application and aggregated. You need both; only the first is APM.
### "What's the difference between Logs, Metrics, Events and Traces?"
The clean way to hold this in your head is: **an event is the atom; the other three
are shapes you give it.**
| | What it is | Cost shape | Answers |
|---|---|---|---|
| **Event** | A structured record of one thing that happened, at one instant, with arbitrary attributes attached | Grows with traffic × attribute count | "What exactly happened here?" |
| **Log** | An event rendered for storage/search. A *structured* log is literally an event; an unstructured log is an event you threw the schema away from | Grows with traffic | "Why did it fail?" |
| **Metric** | A pre-aggregated numeric measurement over many events, sliced by a small set of labels | Grows with *label cardinality*, ~flat in traffic | "Is it broken? How much? Since when?" |
| **Trace** | A causally linked tree of events that each have a *duration* and a parent — a span tree spanning process boundaries | Grows with traffic × span count | "Where did the time go, across services?" |
Three consequences worth internalising now:
1. **Metrics are cheap in traffic but expensive in cardinality.** Doubling your
request rate barely changes Prometheus's memory; adding `user_id` as a label
can kill it. Logs and traces are the opposite: cheap in dimensions, expensive
in volume. This asymmetry drives every design decision later.
2. **Traces are events with two extra fields:** a duration and a parent pointer.
That's the whole trick. Everything else (service maps, critical paths, span
metrics) is derived from those two fields.
3. In OpenTelemetry, "events" aren't a fourth top-level signal you export
separately. They show up as **span events** (a timestamped annotation attached
to a span — this is how `record_exception` works) and as **log records carrying
an `event.name` attribute**; the Events API is built on the Logs data model.
When Babak says M.E.L.T., he most likely means "logs plus the discrete
business/domain events you care about" — check with him, but treat it as
*structured logs and span events*, not a separate pipeline.
### "Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?"
**OpenTelemetry first, without much doubt.**
Loki, Tempo, Prometheus and Mimir are *destinations*. They're databases with query
languages. OpenTelemetry is the **producer-side data model and wire protocol** — it's
the only part of this that touches your Python code, and it's the part that
determines what the destinations can possibly show you. If you learn Loki first you
learn Loki's label model and its promtail-isms, most of which don't transfer. If you
learn OTel first, every backend becomes "point OTLP at a different URL."
One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
early, in parallel with OTel metrics — because it *constrains what you're allowed to
emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
(Phases 4–5), not up front.
Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
part of the Grafana stack and you do not need them for this task. Phase 9 covers
them as a judgment question, which is exactly how Babak framed them.
### Verdict on the OneUptime article you found
**Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
teaching resource. Read it in **Phase 4/5**, not now; it assumes you already
understand spans, span metrics and PromQL.
What's genuinely good in it, and rare to find in one place:
- The `spanmetrics` and `servicegraph` **connectors** in the Collector — this is how
you get RED metrics and a service map "for free" from traces you already emit.
This is the single highest-leverage config in the whole article.
- The Tempo datasource provisioning YAML with `tracesToLogsV2`, `tracesToMetrics`,
`serviceMap` and `nodeGraph` — the click-through correlation that makes Grafana
feel like an APM.
- `exemplarTraceIdDestinations` on the Prometheus datasource — jump from a latency
spike on a graph straight into the trace that caused it.
- The tail-sampling policy block (keep all errors, keep all slow traces, sample the
rest).
Two caveats:
- Its Collector config exports logs via the **`loki` exporter**. Loki now accepts
**OTLP natively** on its own endpoint, and that is the current recommended path —
fewer moving parts, better attribute mapping. Use the native OTLP route.
- It's a vendor blog (OneUptime sells a competing product). The Grafana content is
accurate, but treat the official OpenTelemetry and Grafana docs as the source of
truth for anything spec-level.
- **Its `spanmetrics` / `servicegraph` Collector connectors do not apply to your
stack.** See the next section — your platform generates those metrics in Tempo
instead. The article's PromQL is still useful, but the metric names differ.
---
## The platform Babak gave you (`observability-ansible`)
This changes *where* your telemetry goes and *how the pipeline is configured*. It
does not change the learning arc, the app, or the test suite. Read this before
Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9.
### What the repo is
An Ansible project that deploys **two independent observability backends** on one
Ubuntu/Debian host via Docker Compose:
1. **ELK + Grafana** — Filebeat → Logstash → Elasticsearch → Kibana/Grafana
2. **Grafana stack** — Prometheus + Alertmanager + Loki + Tempo + Grafana, with
**Grafana Alloy** as the collector and node-exporter for host metrics
It contains **no application code and no dashboards**. It is the *platform half* of
Babak's task, handed to you pre-built. Your half is everything that produces
telemetry and everything that reads it.
Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
`playbooks/elk-grafana.yml`, idempotent, versions pinned in `group_vars/all.yml`
(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
Elastic 9.1.3). Everything lands under `/opt/observability/`.
### The architecture
Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
**Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline →
Explore & Correlate**, with three input classes feeding it.
```
App metrics ──/metrics, Prometheus exposition──┐
Container logs ──Docker socket, tail──▶ Alloy ──push API──▶ Loki
Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo
("spans + resource attrs") │
Node Exporter :9100 ──────scrape──────┐ │
▼ │
Prometheus ◀──remote_write────────┘
(TSDB, PromQL, rules) span metrics + exemplars
│
firing alerts ▼
Alertmanager ──route──▶ Receivers
(group, dedupe) (Slack, Email, PagerDuty,
│ webhook, SMTP)
Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four
Correlation key: trace_id / exemplars
```
Note what the diagram says about the OTLP box: **"spans + resource attrs"**. Traces
only. And app metrics are labelled **"/metrics endpoint, Prometheus exposition"** —
not OTLP. The diagram and the code agree on this; it is a design decision, not an
oversight. Plan your instrumentation around it.
### One place the diagram and the code disagree
The diagram draws app metrics as `App metrics → scrape → Alloy → /metrics →
Prometheus`. The shipped config does not do that:
- `config.alloy.j2` contains **no `prometheus.scrape` component at all** — Alloy
collects logs and receives traces, nothing else.
- `prometheus.yml.j2` scrapes `prometheus`, `node-exporter`, `loki`, `tempo`,
`alloy:12345`, `grafana`, then loops over `prometheus_extra_scrape_jobs`. So your
app targets are scraped by **Prometheus directly**, bypassing Alloy entirely. The
green `/metrics` arrow from Alloy to Prometheus in the diagram is Prometheus
scraping Alloy's **own** self-monitoring endpoint, not your app's metrics
passing through it.
Neither is wrong; the diagram shows the intended architecture and the code
implements a simpler one. Closing that gap — adding `prometheus.scrape` +
`prometheus.remote_write` to Alloy so metrics really do flow through the collector —
is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth
mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
exactly the skill this task is meant to build.
### Four things in it that will cost you a day if you don't know them
**1. Alloy's OTLP receiver only forwards traces.** The config is literally:
```
otelcol.receiver.otlp "default" {
grpc { endpoint = "0.0.0.0:4317" }
http { endpoint = "0.0.0.0:4318" }
output {
traces = [otelcol.exporter.otlp.tempo.input]
}
}
```
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
your app sends are accepted and then **silently discarded** — no error, no warning,
just nothing arriving in Grafana. This is the single most likely thing to eat an
evening.
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
input. So you have three options, and you should understand all three:
- *Follow the platform's grain (simplest):* expose a Prometheus `/metrics` endpoint
per service and register them in `prometheus_extra_scrape_jobs` in
`group_vars/all.yml`. Prometheus scrapes your services directly. Logs continue to
arrive via stdout scraping.
- *Implement what the diagram draws:* add `prometheus.scrape` +
`prometheus.remote_write` components to Alloy so app metrics flow through the
collector, as the architecture intends. Buys you central relabeling and one place
to change collection policy.
- *Go all-OTLP:* add `metrics` and `logs` outputs to the receiver, plus
`otelcol.exporter.prometheus` → `prometheus.remote_write` and
`otelcol.exporter.loki` → `loki.write`. One protocol for all three signals — the
architecture you'd argue for on a greenfield platform, and the furthest from what
ships.
Do the first to get moving, then do at least one of the others as a deliberate
exercise and write up the trade-off. That comparison is worth more than any single
answer.
**2. Span metrics come from Tempo, not from the collector.** `tempo.yml.j2` enables:
```
metrics_generator:
storage:
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true
overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics, local-blocks]
```
So RED metrics, the service graph, and exemplars already exist — generated inside
Tempo and remote-written into Prometheus (which runs with
`--web.enable-remote-write-receiver`). **Do not add `spanmetrics`/`servicegraph`
connectors to Alloy** — you'd be generating the same series twice under different
names. The metric names differ from the OneUptime article's; go find the actual
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.)
**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
datasource is provisioned with:
```
derivedFields:
- datasourceUid: tempo
matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
name: TraceID
```
Your log lines must contain **lowercase** `trace_id` (or `traceid` / `trace-id`).
This matters more than it looks: OpenTelemetry's `opentelemetry-instrumentation-logging`
auto-injection emits **`otelTraceID`**, which does **not** match that pattern.
So on this platform, the structlog processor emitting a lowercase `trace_id` field
isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus
datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram states
this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
in the Explore & Correlate column hangs off that one field name.
**4. There is no dashboard provisioning at all.** The Grafana provisioning mount
contains only `datasources/`. Every dashboard is yours to add — which means Babak's
"create dashboards as code, destroy and recreate in seconds" requirement is entirely
your deliverable, and the natural home for it is *inside this Ansible role*.
### What this changes, phase by phase
| Phase | Impact |
|---|---|
| 0 / 0a | **None.** App and test suite are unaffected. |
| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
| 2 | Endpoint changes to Alloy; tracing work itself is identical. |
| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
| 6 | None. |
| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
| 8 | None, plus TraceQL metrics from `local-blocks`. |
| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
| 10 | Demo now includes `ansible-playbook` rebuilding the platform. |
### Where to run it
The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside
everything else will hurt. Use a separate VM (a cheap cloud instance or a local
Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing
at it. That also makes the deployment *real* rather than a localhost shortcut, which
is the point of it being Ansible.
If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker
socket and will scrape **every** container's stdout, including Ledgerline's,
automatically — convenient, and worth knowing before you wonder where those logs
came from.
### One thing to confirm with Babak
Is this repo a **platform you must consume as-is**, or a **starting point you may
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
extend `group_vars`. If Babak intends it as a fixed platform, several
recommendations below change — ask before Phase 3.
---
## Your resource spine
Pick **one paid course as a backbone** and use the free material as targeted
supplements. Both Udemy options below ship captions, which matters for your
subtitle-to-notes workflow.
**Backbone — choose one:**
| Resource | Format | Length | Cost | Best for |
|---|---|---|---|---|
| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. |
| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. |
If budget allows, buy both — they overlap by maybe 30% and the second half of each
is different. If only one: take the **OpenTelemetry** one, because instrumentation
is the part you have to *write*, and Grafana's own docs are excellent for the rest.
**Free, and worth your time:**
| Resource | Format | Length | Use in |
|---|---|---|---|
| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. |
| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. |
| **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. |
| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 |
| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. |
| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on |
| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 |
| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. |
| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 |
| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. |
| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 |
| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 |
**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on
your k8s cluster until everything works locally — debugging instrumentation and
debugging Kubernetes at the same time is how these projects die. Porting it to the
cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half
Babak has planned for you.
docs/roadmap/phase-0.md
+127
View File
@@ -0,0 +1,127 @@
# Phase 0 — Build the base project, test-first
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`intro.md`](intro.md) · Next: [`phase-1.md`](phase-1.md)*
**Status:** Done
**Message:** Both halves complete and verified against the live stack, not just
collected.
- **0a (test suite):** 81 tests across `tests/` (11 files + a 5-module
harness), covering all eleven goals G1–G11 and all 13 named pathologies.
Verified genuinely red against the empty repo before any implementation
existed; `tests/RED-BASELINE.md` records that run. `tests/README.md`
documents the invented request-contract fields, measurement methodology,
and known-flaky tests.
- **0b (implementation):** all seven services (`gateway`, `ledger`, `fraud`,
`sanctions`, `insights`, `notifier`, `audit`), `db/schema.sql` +
`optional_indexes.sql`, `scripts/seed.py` + `smoke.py`, the frontend, and
`docker-compose.yml`. Seeded: 800 accounts, 250,000 transactions, 45
merchant categories, 12,000 sanctions entries, in ~12s. `make smoke` and
`make test-contract` (64/64) pass; `make test-baseline` and
`make test-pathology` pass except for a handful of tests documented below
as load-sensitive rather than broken.
- **Two changes to the frozen test contract, both made only after flagging
them and getting an explicit go-ahead — not applied unilaterally:**
- `test_row_counts_match_manifest_and_targets` loosened the `transactions`
row-count check from exact equality to `>=`, since other contract tests
in the same `make test-contract` run legitimately insert rows first.
- `test_pool_starvation_plateaus_throughput` was **removed outright**.
Its baseline assertion (throughput scales ≥2.5x from concurrency 5→25)
is incompatible with this same spec's own §3 mandate of one uvicorn
worker per service — confirmed by reproducing the identical
non-scaling behaviour on a bare `GET /healthz` with zero backend
logic. `pool_starvation` itself is still fully implemented
(`services/ledger/db.py`'s dual-engine pool selection); only that one
differential test was dropped.
- **Known-flaky, documented rather than papered over:**
`test_clean_baseline_is_healthy`, `test_nightmare_degrades_but_survives`,
the baseline half of `test_blocking_event_loop_serializes_concurrent_requests`,
and `test_missing_index_forces_seq_scan` compare a fast-path measurement
against a slow-path one under real concurrent load, and pass at roughly a
60–80% rate on ordinary dev hardware depending on momentary host
contention. See "Known-flaky tests" in `tests/README.md`.
- **Notable implementation deviations** (none change observable behaviour of
the 13 pathologies or the request/response contract): gateway's `GET /`
serves the frontend rather than §4's generic info JSON (the other five
HTTP services still return that JSON as specified); `insights` splits a
cheap PK-only `quick_summary` (embedded in every transaction response)
from the real cached/N+1-affected 120-day analytics endpoint, so the
missing-index pathology doesn't tax every request; `retry_storm` fires
all 4 attempts unconditionally rather than stopping at first success
(spec's "no circuit breaker" taken literally — also the only way to reach
the required lookup-multiplication ratio given `sanctions_errors`' fixed
3% rate); `poison_data` resolves via a Redis hash populated at seed time
instead of a live table scan, to avoid the deliberately missing index;
uvicorn's access log is disabled fleet-wide (redundant with the spec's
own logging design, and was double-counting two log-line-based pathology
tests); Postgres `max_connections` raised from 100 to 300 (five
services' pool sizes can sum past the default under load).
**Time-box:** 2–3 evenings. Two agent sessions plus one evening of your own
reading.
**Deliverable:** the working Ledgerline system with no observability, plus a
black-box integration suite that pins down both its working state and its
deliberate brokenness.
### Why TDD here, when the app itself doesn't matter
The microservices are throwaway. The *properties* aren't. Everything from Phase 2
onward assumes the system still behaves a specific way: seven real services in
the path, async work genuinely off the request path, five hot accounts owning a
third of the rows, and thirteen pathologies that actually degrade something. You
will be editing all seven services for weeks. Without a suite that fails when one
of those properties quietly breaks, you'll spend an evening chasing a trace
anomaly that turns out to be a regression you introduced yesterday.
Writing the tests first also forces the contract to be explicit before anything
is built — which is what makes a second agent able to implement against it
without guessing.
### 0a — the test suite
Agent session one, from `phase-0a-test-suite-spec.md`. That agent writes only
`tests/` and must not create so much as a stub under `services/`. Its output
should be **red**, with a `tests/RED-BASELINE.md` recording why each test fails
right now — a test that passes against an empty repo is a broken test.
The suite asserts goals, not mechanics. Two representative examples worth
understanding before you read the code:
- *"Stop the `sanctions` container and transactions must fail."* This is the test
that catches an implementation quietly collapsing seven services into one
process. Nothing else would.
- *"With `n_plus_one` on, one insights request issues ≥ 50 SQL statements."*
Measured via `pg_stat_statements`, not wall clock — because a query count
can't be faked with a `sleep()`, and a latency threshold can.
That second point is the design principle throughout: **prefer assertions that
are hard to satisfy without the real mechanism.** Query counts, `EXPLAIN` plan
node types, Redis consumer-group lag, container RSS, and genuine unhandled
exception paths all have that property.
### 0b — the implementation
Agent session two, from `phase-0-base-project-spec.md`, with `tests/` handed over
as a **read-only contract**. That agent may not edit, skip, `xfail` or loosen a
single test. If something looks impossible it stops and reports rather than
adjusting the target.
Done when `make test` is green three consecutive runs and
`git diff --stat tests/` is empty.
### Then, yourself
1. **Read every service's `app.py` end to end.** You need to know this codebase
cold, because from Phase 2 onward you're modifying all seven services.
2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
then try to figure out what's wrong using only `docker compose logs`.** Give it
twenty real minutes. You will fail, and the specific way you fail — no timing,
no causality, no cross-service correlation, a `X-Request-Id` that dies at the
Redis Stream — is the motivation for everything that follows. Write down what
you wished you had. That list is your acceptance criteria for Phase 8.
**Done when:** `make test` is green three runs in a row, all nine acceptance
criteria in the base spec pass, and you have that written list of frustrations.
docs/roadmap/phase-1.md
+89
View File
@@ -0,0 +1,89 @@
# Phase 1 — The model, and one signal end to end
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
**Status:** Not started
**Message:** Blocked on nothing — Phase 0 is done and the platform is
described above. Waiting to begin.
**Time-box:** one evening.
**Why:** you need the vocabulary, but more importantly you need to see one trace
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
than abstract.
### Concepts
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
(known unknowns): dashboards and alerts built from predefined aggregates.
Observability is a property of the system: can you ask a *new* question about
production, that you never anticipated, without shipping new code? Charity Majors'
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
practical implication for you: emit **wide, high-dimensional events** (rich span
attributes), not just pre-aggregated counters.
- **The OTel architecture split: API vs SDK vs Collector.**
- The **API** is what your code and third-party instrumentation libraries call.
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
library can be instrumented without forcing a dependency on any implementation.
- The **SDK** is the engine: sampling, batching, context propagation, export.
Application owners configure the SDK; library authors only touch the API.
- The **Collector** is a standalone process that receives, processes and exports
telemetry. Optional, but you should use one.
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
`service.version`, `deployment.environment`) and is attached to every signal from
that process. Attributes describe the individual span/metric/log.
- **Semantic conventions**: standardised attribute names (`http.request.method`,
`db.system`, `server.address`). Use them for anything standard; namespace your own
(`ledgerline.account.id`). This is what makes cross-service, cross-language
correlation work — and it's what Grafana's prebuilt dashboards key off.
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
aware of, not learning yet.)
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
seven services. The Collector also does the things you don't want in your app:
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
destinations.
### Resources
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
M.E.L.T. and context-propagation sections; skip the Node code.
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
it — just see what "done" looks like.
### Build
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
`ansible-galaxy collection install -r requirements.yml`, change
`grafana_admin_password` (vault it), then
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
Then open the Alloy UI and **read the component graph** — it shows you the exact
pipeline, including the fact that the OTLP receiver has only a traces output.
Seeing that in the UI before you get bitten by it is worth ten minutes.
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
a single container with everything pre-wired. Useful for a first trace, but
don't build on it — its pipeline differs from your real platform in exactly the
ways that matter.)*
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
auto-instrumentation:
```
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```
and change its command to
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
with `OTEL_SERVICE_NAME=sanctions`,
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
Tempo directly.
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
**Done when:** you can see a single-service trace in Tempo, and you can explain out
loud what the Resource attributes on it mean and where they came from.
docs/roadmap/phase-10.md
+40
View File
@@ -0,0 +1,40 @@
# Phase 10 — Package the delivery
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)*
**Status:** Not started
**Message:** Depends on every prior phase being done — this is the wrap-up,
not independent work.
**Time-box:** one evening.
Babak gave you a task; deliver it like an engineer, not like a student.
1. **`README.md`** — architecture diagram, one-command start, what each container is
for, and a "here's the demo" script.
2. **The five-minute demo path**, rehearsed:
- Delete every dashboard in the Grafana UI, then `ansible-playbook
playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
from code, one command.
- Submit one transaction from the frontend, then follow it through Grafana: RED
dashboard → latency spike → exemplar → trace → span → logs for that span → back
to the trace. One request, all three signals, no searching.
- `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
and the service map turn red at a specific node.
- Show `docs/INVESTIGATION.md` and how you found the pathologies.
3. **The written answers** Babak asked for, as documents in the repo, not as a
conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
`docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
RTO/RPO/SLA/SLO/error budget in your own words.
4. **The test suite as part of the story.** Show `make test` green, then show the
three generations of the call-counting assertion side by side —
`docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
each signal bought you. Most people demo dashboards; almost nobody can show
their observability work paying off inside their own test suite.
5. **A "what I'd do differently in production" section.** Tail sampling instead of
100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
you *didn't* do, and why, is the strongest signal of seniority in the whole
delivery.
docs/roadmap/phase-2.md
+116
View File
@@ -0,0 +1,116 @@
# Phase 2 — Distributed tracing
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
**Status:** Not started
**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
(Resource, providers, exporters, Collector wiring) is shared across all three
signals — do it once here and logs and metrics become incremental. Second, tracing
is what Babak actually asked for ("trace a single request, see where it stuck"),
and once trace IDs exist, log correlation in Phase 3 works on the first try instead
of needing rework.
### Concepts
- **Trace / span / span context.** A trace is one request's journey; a span is one
timed operation within it. Each span carries a trace ID, its own span ID, and its
parent's span ID. That parent pointer is the entire data structure.
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
these right is what makes the service graph render correctly.
- **Context propagation.**
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
carry the context explicitly.
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
- The classic bug: any boundary that doesn't carry those headers **starts a brand
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
producer and `.extract(...)` on the consumer, then starting the consumer span
with that context as parent. This is the single most valuable exercise in the
phase — most people never do it manually and never really understand it.
- Related: `links` (for a consumer span that batches messages from many traces)
and **Baggage** (arbitrary key/values propagated alongside trace context —
useful, and a foot-gun, because it's sent on every hop and is visible to
downstream systems; never put secrets in it).
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
that show *which service* was slow but not *why*.
- **Span attributes vs metric labels.** Spans are the right home for
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
with volume, not with distinct values. This is why "put it on the span, not the
metric" is the standing rule.
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
That's how the exception ends up attached to the trace and how the `spanmetrics`
connector in Phase 4 will count your error rate.
- **Sampling — the concept now, the config later.**
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
errors at the same rate as successes.
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
traces, but must run in the Collector and buffer spans in memory.
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
now; you'll configure tail sampling in the Collector in Phase 5.
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
unilaterally drop half of a trace its caller decided to keep.
### Resources
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
practical FastAPI-specific setup, including batch/queue env vars.
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
reading.
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
side by side. Look specifically at how they mix auto-instrumentation with
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
This is the "professional implementation" you asked to study.
### Build
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
service calls it once at startup. Keep it under 60 lines.
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
`ledgerline.tx.amount_minor`).
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
inject trace context into the message fields. Consumers (`notifier`, `audit`):
extract, and start a `CONSUMER` span with the extracted context as parent.
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
more.
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
the "before" state so you could measure against it, and from here it's
actively wrong. Everything else in `tests/` must still pass: instrumenting
the system is not allowed to change its behaviour, and if `make test` goes red
after adding spans, you've broken something real. That is exactly the safety
net you built it for.
### Done when
- One request produces **one connected trace** covering all seven services,
including both async consumers.
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
ate the time.
- You can explain why `blocking_event_loop` looks different in a trace than
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
before spans start, plus every concurrent request in that process degrading
together.)
docs/roadmap/phase-3.md
+99
View File
@@ -0,0 +1,99 @@
# Phase 3 — Structured logs, correlated to traces
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
**Status:** Not started
**Message:** Depends on Phase 2 (trace IDs must exist before log correlation
means anything).
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
### Concepts
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
events: a stable event name plus typed fields (`event="transaction_settled"`,
`account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
Consistent field names across all seven services — `account_id` everywhere, never
`accountId` in one place and `acct` in another. This is what makes LogQL queries
possible at all.
- **Trace–log correlation.** Every log line emitted inside an active span must carry
`trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
"one pane of glass": click a span → see its logs; click a log line → open its
trace. Both directions are configured, and you'll set both up.
- **Two ways to ship logs, and which one your platform actually does:**
- **stdout → collector scraping → Loki.** The container-native way. Your app just
prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
standard in Kubernetes. **This is the only path wired in your Alloy config**
(`loki.source.docker` reading the Docker socket).
- **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
with the same Resource attributes attached automatically. Cleaner correlation,
but the app owns delivery. **On your platform this currently goes nowhere** —
Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
dropped in silence.
- Do both anyway: use stdout for most services, and *extend Alloy* to wire the
OTLP logs path for one or two. Then argue the trade-off from experience. That
comparison — including the fact that a misconfigured pipeline fails silently
rather than loudly — is precisely what Babak will probe.
- **The field name is not a style choice here.** Your Loki datasource is provisioned
with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
A log line must carry lowercase `trace_id`. OpenTelemetry's logging
auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
logs and traces both arriving, no errors anywhere, and no clickable link. Hence
structlog with an explicit processor rather than the auto-injection shortcut.
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
your entire fleet, and Loki bills/indexes on volume.
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
`json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
redaction into the processor chain, not into each call site.
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
not log content. Labels should be few and low-cardinality (`service`, `env`,
`level`). Everything else lives in the log line body and is filtered at query time.
Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
in the JSON body and use a **derived field** to make it clickable.
### Resources
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
setup, including the hex formatting for trace/span IDs.
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
`OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
fields, if you'd rather not hand-roll a processor.
- Udemy backbone: the Loki + LogQL sections.
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
### Build
1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
processor that pulls the current span context and injects `trace_id` / `span_id`.
2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
handler: request received (or rely on the span), business decision, error.
3. Add a redaction processor with a denylist of field names.
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
with a `container` label (Alloy's relabel rule sets it from the Docker container
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
the receiver shows traffic, and the pipeline shows no logs output. Now extend
`config.alloy.j2` — add a `logs` output on the receiver plus an
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
That is the most transferable half-hour in the whole project.
6. Confirm the provisioned correlation works in both directions: Loki's
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
are verifying, not writing. If a link doesn't appear, the regex didn't match
your field name — check that before anything else.
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
(`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
in `docker compose logs` — the only cross-service call counter that existed at
Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
with LogQL over structured fields instead. This is a small change and a
pointed one: notice how much shorter and more precise the assertion gets, and
that it now works without shelling out to Docker at all. That delta is the
value of structured logging, measured on your own code.
**Done when:** from a slow trace you can click into that request's logs, and from
any log line you can click into its trace — in both directions, in two clicks.
docs/roadmap/phase-4.md
+126
View File
@@ -0,0 +1,126 @@
# Phase 4 — Metrics
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
**Status:** Not started
**Message:** Not blocked technically (metrics don't strictly need tracing
first), but sequenced after Phase 3 per the roadmap's ordering.
**Time-box:** 2 evenings.
### Concepts
- **Instrument types.**
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
connection-pool checkouts.
- **Histogram** — a distribution. Request latency, payload size. This is what
gives you percentiles.
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
read via a callback on collection (right for things you *poll*, like pool size).
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
nonsense. Use histograms.
- **Cardinality — the one thing that will bite you.** Every unique combination of
metric name + label values is a separate time series in memory. Add `account_id`
as a label on a 800-account demo and you've made 800 series; do it in production
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
~10 000 distinct values in production, it does not belong on a metric.** Put it on
a span. Histograms multiply this (one series per bucket per label combination), so
don't over-bucket either.
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
the concrete path), `http.status_code` class, `kind`, `currency`.
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
raw URL, timestamp, error message.
- **RED and USE.**
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
per-service dashboard.
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
connection pools, queues, disk. Saturation is the one everyone forgets and the
one that predicts collapse.
- You want both: RED tells you the service is slow, USE tells you which resource
is why.
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
and knowing both is the point:
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
`servicegraph` connectors read spans in flight and emit RED metrics and edge
metrics. This is what most blog posts (including the OneUptime article) show.
- *In the trace store:* Tempo's **metrics-generator** does the same job after
ingestion, remote-writing the result to Prometheus. **This is what your platform
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
`send_exemplars: true`, remote-writing to a Prometheus started with
`--web.enable-remote-write-receiver`.
- So RED metrics, the service map and exemplars **already exist** before you write
a single metric line. Do not add the collector connectors as well — you'd
duplicate every series under a second set of names. **Go find the real metric
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
they will not match the article's.
- The trade-off worth being able to argue: collector-side generation is cheaper
and works before storage, but sees only what passes through that collector;
Tempo-side generation sees everything that landed, survives collector restarts,
and can be enabled per-tenant — at the cost of load on the trace store.
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
click a spike on a latency graph and land in the exact trace. On your platform
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
`name: trace_id`. If you emit your own histograms with exemplars, the label must
be `trace_id` to match.
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
the pull model is why Prometheus has service discovery and why `up` exists.
### Resources
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
Read the first half. Best cardinality explanation available.
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
not a fact.
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
`servicegraph` connector config and its RED PromQL queries almost verbatim.
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
### Build
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
2. Custom application metrics, deliberately few:
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
- `ledgerline.fraud.score` (Histogram; no labels)
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
metrics, so pick a path and implement it:
- *Platform grain:* expose `/metrics` on each service and add them to
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
playbook. Prometheus scrapes your services directly. Note the irony and the
lesson — you're back to a pull model and a second client library, which is
exactly the fragmentation OTel exists to fix.
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
Alloy so app metrics flow through the collector as the architecture drawing
intends. This also closes a real diagram-vs-code gap in the repo.
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
all three signals.
Do the first to get moving; do one of the others afterwards and write up which
you'd choose for a real platform team and why. That write-up is a Phase 9
artefact.
4. Deliberately break it: add `account_id` as a label to
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
you'll never forget.
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
one. Understand why they differ slightly and when you'd want each.
6. **Second migration of the call-counting assertions.** The three tests you moved
to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
TraceQL query for `sanctions` client spans within a single trace answers
"how many lookups did this one transaction make?" directly, which is the
question the test was always really asking. Do it and compare all three
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
clearest possible demonstration of what each signal is actually for, on a
problem you already understand.
**Done when:** you have a RED view per service, a working service map, and you can
click a latency spike through to a trace via an exemplar.
docs/roadmap/phase-5.md
+76
View File
@@ -0,0 +1,76 @@
# Phase 5 — Dashboards and alerts as code
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
**Status:** Not started
**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
dashboarding.
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
recreate in seconds.*
### Concepts
- Three levels of "as code", and you should know when each applies:
1. **File provisioning** — YAML in `provisioning/datasources/` and
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
container. Grafana loads them at startup. Simplest, no state, no API tokens,
perfect for a Compose demo. **This alone satisfies the requirement.**
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
of Grafonnet (which is legacy and unofficial).
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
when this lands on your Kubernetes cluster.
- **Dashboards should answer questions, not display data.** A good service dashboard
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
is the pressure. Panels that nobody looks at during an incident are cost, not value.
- **Alerts as code too.** Grafana alert rules, contact points and notification
policies all provision from YAML. And keep the *alert definition* next to the
dashboard that explains it.
### Resources
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
example.
- Udemy backbone: dashboard design sections (panel types, transformations,
variables, dashboard best practice).
### Build
Your platform provisions **datasources only** — there is no dashboards provider at
all. So this phase is entirely yours, and the right home for it is inside the
Ansible role, using the same mechanism as everything else.
1. **Extend the role.** Add
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
(a file-based dashboard provider pointing at a mounted directory), a
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
and copy them. The Compose template already mounts
`./config/grafana/provisioning` read-only, so the provider is picked up on
restart.
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
Foundation SDK (Python)**: one function producing a RED dashboard for a given
service name, looped over the seven services; a second producing a system
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
the generator *and* its output; the Ansible role ships the output.
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
than a local Makefile target, because it's the same command that builds the
platform.
4. While you're in there, review the provisioned datasources and add what's missing:
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
understand why `filterByTraceID: true` alone is sufficient here, and when it
wouldn't be.
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
**Done when:** re-running the playbook restores every dashboard from scratch, and no
dashboard has ever been saved through the Grafana UI.
docs/roadmap/phase-6.md
+94
View File
@@ -0,0 +1,94 @@
# Phase 6 — Load testing with k6, and percentiles
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
**Status:** Not started
**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
benefits from Phase 5's dashboards existing.
**Time-box:** 1–2 evenings.
### Concepts — P95 and P99 properly
You asked about these specifically, and there's more to them than "the 95th
percentile."
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
is.
- **You cannot average percentiles.** Not across instances, not across time buckets.
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
to aggregate the *histogram buckets* and compute the quantile from the sum:
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
bracket your actual SLO threshold. (Native histograms largely solve this and are
worth knowing about.)
- **Which population?** An overall p99 across all endpoints is dominated by whichever
endpoint has the most traffic. Slice by route, or you'll miss that
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
- **Tail latency amplification** — the most important idea here for a microservice
system. If one user request fans out to 10 backend calls, each with an independent
p99 of 1 s, then the chance the *user request* touches at least one slow call is
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
measurable — go measure it.
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
that.
- **Coordinated omission** — a load-generator trap. If your generator waits for a
response before sending the next request, then when the system stalls, the
generator stalls too and simply *doesn't record* the requests that would have been
slow. Your measured latency is systematically optimistic. Fix: use an **open
model** — a fixed *arrival rate* independent of response times. In k6 that means
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
`constant-vus`. Knowing this is a genuine senior-level differentiator.
### Concepts — k6
- **Test types:** smoke (does it work at all), average-load (normal traffic),
stress (past normal, find the knee), spike (sudden surge, test recovery),
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
fail the run; **thresholds are SLOs expressed in code and they set the process exit
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
view and the application's view sit on the same Grafana time axis. That side-by-side
is the single most convincing screenshot you can show Babak.
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
writing 60 lines of it.
### Resources
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
*Thresholds*. Free, genuinely well written; ~2 h to work through.
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
hands-on).
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
- Grafana docs → *k6 output to Prometheus remote write*.
### Build
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
sharing a `common.js` for setup and payload generation against random accounts.
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
`http_req_failed rate<0.01`.
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
sets. This before/after pair is your baseline for Phase 8.
6. Optional but impressive: have k6 send W3C trace context headers so load-test
requests appear as traces, letting you click from a k6 spike into a specific
trace. Check the current k6 docs for the tracing module's status — it has moved
between experimental and stable.
**Done when:** `make load-stress` produces a Grafana view where you can see the
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
service that broke first.
docs/roadmap/phase-7.md
+91
View File
@@ -0,0 +1,91 @@
# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
**Status:** Not started
**Message:** Depends on Phase 6's load-test baseline (the latency SLO should
come from measured numbers, not be picked from thin air).
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
### Concepts
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
`POST /api/transactions` requests that returned 2xx within 500 ms.*
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
requests succeed within 500 ms, measured over a rolling 28 days."
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
Always **looser** than your SLO, so you have room to miss internally without
breaching commercially. If your SLA and SLO are the same number, you've made a
mistake.
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
budget; over 30 days that's about **43 minutes**. The budget is the thing that
turns reliability from an argument into arithmetic: if there's budget left, ship
features and take risks; if it's spent, the policy says you stop feature work and
spend on reliability. That policy — agreed in advance by dev, ops and product — is
the whole point.
- **Burn rate.** How fast you're consuming budget relative to the window.
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
is what stops you from either paging on every blip or noticing the slow bleed a
month late.
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
and it's worth being precise because interviewers probe exactly here.
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
from failure to restored service. It drives your architecture: replication,
warm standby, failover automation, runbook quality.
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
in time: how far back can you afford to lose. It drives your **backup and
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
every 15 minutes.
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
5 minutes of transactions." Both feed the SLA.
- Both are **business decisions, not technical ones** — you derive them by asking
what an hour of downtime or an hour of lost data actually costs, then buying the
cheapest architecture that meets it. Tightening either is expensive and
non-linear.
### Resources
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
all free PDFs. Run the exercises solo with Ledgerline as the subject.
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
free, self-paced, deeper.
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
overview; pair it with your Postgres backup knowledge.
### Build
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
- Availability SLI/SLO on `POST /api/transactions`
- Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
from thin air)
- A freshness/lag SLI on the notifier consumer group
- A stated SLA that is deliberately looser than each SLO
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
- An error budget policy: what happens at 50%, 75% and 100% consumed
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
be able to argue the alternative: Grafana unified alerting can evaluate across
datasources (including Loki queries) and lives with your dashboards, whereas
Alertmanager is metrics-only but is the ecosystem standard and gives you
grouping, deduplication, inhibition and silences for free. Note which you chose
and why in `docs/SLO.md`.
3. Add an error-budget panel to the overview dashboard: budget remaining, current
burn rate, projected exhaustion.
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
watch the fast-burn alert fire.
**Done when:** you can explain, in one paragraph each and without notes, the
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
decision.
docs/roadmap/phase-8.md
+90
View File
@@ -0,0 +1,90 @@
# Phase 8 — Bottleneck hunting (the capstone)
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
**Status:** Not started
**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
**Time-box:** 2 evenings. This is where the whole thing pays off.
### Concepts — how to actually answer "how do I find bottlenecks?"
Babak asked how you'd *think* about this, not for a list of tools. Here's the
method, roughly in order. The discipline is to go **top-down from the user, not
bottom-up from the infrastructure** — bottom-up is how people spend three hours
tuning a database that wasn't the problem.
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
2. **Localise to a service** using RED + the service graph. Which node's duration
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
the graph from the leaves inward.
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
several slow traces plus one fast trace of the same endpoint as a control.
4. **Read the trace for five specific shapes:**
- **A long span** — the work itself is slow. Look at its attributes: which query,
which host, which parameters.
- **A gap** — wall-clock time not covered by any span. This is the interesting
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
blocked event loop, or a missing instrumentation library. Ledgerline's
`pool_starvation` and `blocking_event_loop` both show up as gaps.
- **Serialisation** — spans laid end to end that could have been concurrent.
Ledgerline's `no_batching` looks exactly like this.
- **Repetition** — the same span name 150 times in a row. That's your N+1.
- **Retries** — the same client span repeated 3–4 times with errors. That's
`retry_storm`, and note that it makes an *overload* worse, never better.
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
memory. A resource at 100% utilisation with a growing queue is saturated;
latency there is a *symptom*, not the cause.
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
concurrent slots. If your pool is 5, you queue — and queueing latency grows
non-linearly as you approach saturation. This one formula explains most
connection-pool and worker-count incidents you'll ever see.
7. **Check for tail amplification** (Phase 6) before blaming any single service.
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
over the *same duration*. One variable at a time. Never declare a fix from a
dashboard that "looks better."
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
becomes the bottleneck. Re-measure after every change; stop when you're inside
the SLO, not when you run out of ideas.
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
or lock waits inside Postgres, or which line of Python burned the cycles. That's
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
### Resources
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
for saturation.
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
thing ever written on why microservice latency behaves the way it does.
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
these; connect them to the trace's DB spans.
### Build
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
`docs/PATHOLOGIES.md`.**
2. Run `make load-average` for 10 minutes.
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
like a real incident doc: what you observed, what you hypothesised, what you
checked, what it turned out to be. Include screenshots.
4. When you think you've found them all, run `make chaos-clean`, enable them one at
a time, and confirm each signature.
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
6. Repeat once with a new random set. The second run is where the method actually
becomes yours.
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
docs/roadmap/phase-9.md
+161
View File
@@ -0,0 +1,161 @@
# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
**Status:** Not started
**Message:** Not technically blocked by earlier phases (it stands on its own
as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
re-diagnose in Kibana already exists.
**Time-box:** 1–2 evenings.
Babak's remaining questions are all *judgment* questions. He isn't testing whether
you can recite what Filebeat does; he's testing whether you have a framework for
choosing. Here's the framework, then the answers as worked examples.
### The framework — six axes to reason on
Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
1. **Where does the data originate?** Inside your application (you can instrument it)
or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
2. **What shape is it already in?** Structured JSON at the source, or unstructured
text needing grok/regex parsing? Parsing cost has to land *somewhere*.
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
4. **What's the destination, and how many?** One backend forever, or several, or one
you expect to change?
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
on every host) distributes cost and adds resilience but multiplies config surface.
Parsing centrally concentrates cost and creates a single point of failure — but
one place to change rules.
6. **What already exists, and what does it cost?** Existing Elastic investment,
existing licences, team skills, and licensing constraints (Elastic's Filebeat and
Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
Bit and Vector are permissively licensed). Migration is never free.
### The answers
**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
from a host or container and forwards them with minimal transformation — and your
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
included. Reach for it when the source is a file you don't control the format of, the
app can't be modified, and Elasticsearch is where it's going.
**When to use Logstash.** When you need real *transformation*: parsing unstructured
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
splitting or merging events, conditional routing to multiple destinations, or
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
per host.
**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
architecture and the reason is a **division of labour between edge and centre**:
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
backpressure, ships onward. You want the thing running in a thousand places to be
small and boring.
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
you want them in a handful of places you can redeploy, not on a thousand hosts.
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
persistent queue absorbs it and Filebeat backs off gracefully, rather than every
host queuing independently or dropping.
So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
need central buffering. Skip Logstash when your logs are already structured JSON and
Elasticsearch ingest pipelines can do the little parsing that remains.
**When to use OpenTelemetry.**
- You own the application code and want telemetry generated *from inside it* — spans
with business context, custom metrics, logs already carrying trace IDs.
- You need **more than one signal**, correlated. This is the strongest argument: one
SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
threaded through all three. Neither Filebeat nor Logstash does traces.
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
data model across all of them.
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
Jaeger by editing Collector YAML.
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
rewriting — without touching the app.
**When *not* to use OpenTelemetry.** This is the question that separates people who
have used it from people who have read about it.
- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
file-based collector.
- **You only need one signal from one source with no correlation.** A single nginx
access log going to a single log store: OTel is more machinery than the job needs.
Fluent Bit or Filebeat is simpler and finishes the job today.
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
Elastic estate with hundreds of Beats modules, dashboards and detection rules is
years of investment. "We should use OTel because it's the standard" is not a
business case. Note the direction of travel though: the Collector can already
receive from and export to most of these, so a *gradual* migration is usually the
right shape — often by putting a Collector or Vector in front of the existing
agents rather than ripping them out.
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
retention, different query patterns, different cost model.
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
matured at different rates per language, and profiling is still alpha. Check the
status page for your language before betting on a signal.
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
on it. A two-person team that needs error tracking by Friday should install Sentry.
The senior framing to give Babak: *these tools aren't competitors so much as
different layers.* OpenTelemetry is a **generation and transport standard** for
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
Elasticsearch are **stores**. Most real architectures use one from each layer, and
the interesting question is always which layer a given problem lives in.
### Resources
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
read these to see how the Collector does the Filebeat/Logstash job itself.
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
caveats.
### Build — and this is no longer hypothetical
Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
Filebeat reading Docker container logs and Logstash parsing JSON into daily
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
questions — the codebase he handed you contains both worlds.
1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
Alloy → Loki. Compare them on the same data.
2. Test the direct ingest path the README documents — pipe a JSON line into
Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
parsing happens before you change it.
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
that parses them. Then do the same job with Alloy's `loki.process` stages. Time
both. Note which one you'd rather debug at 2 a.m.
4. Compare the *investigation experience* directly: take one incident you already
diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
for some questions, and Loki's cost model genuinely beats Elasticsearch's for
others.
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
permissively licensed. For some employers that alone decides it.
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
valuably — what you actually found annoying about each. Having run both stacks
against the same application is the difference between reciting a comparison and
arguing one.
docs/roadmap/summary.md
+71
View File
@@ -0,0 +1,71 @@
# Roadmap summary — all phases at a glance
*Part of `docs/roadmap/` — the one-page overview. Read [`intro.md`](intro.md)
first for platform/background context; read the individual `phase-N.md` file
for the phase you're actually working on. This file exists so an agent (or
you) can check overall status and sequencing without loading every phase's
full detail.*
Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase
file follows the same shape: **Status**, **Message** (what's done, notes,
warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
## Status and focus
| Phase | File | Status | Focus | Time-box | Main output |
|---|---|---|---|---|---|
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
## What each phase is, in one line
- **0 — Build the base project, test-first.** Write the black-box test suite,
then implement the seven-service system against it as a frozen contract.
**Done** — see `phase-0.md` for the full implementation report.
- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
Ansible and get one auto-instrumented trace into Tempo.
- **2 — Distributed tracing.** The heart of the task: auto- and
manual-instrument all seven services, and hand-propagate trace context
across the Redis Stream hop, which carries none by default.
- **3 — Structured logs, correlated to traces.** Replace prose logging with
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
in both directions.
- **4 — Metrics.** Instrument RED metrics per service, understand cardinality
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
silently. Tempo's metrics-generator already produces span metrics and the
service graph — don't duplicate it in the collector.
- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the
Grafana Foundation SDK and ship it through the Ansible role, so "destroy
and recreate in seconds" is a real, repeatable command.
- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't
be averaged, what coordinated omission is, and drive Ledgerline with
arrival-rate executors to get a clean before/after load profile.
- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
SLOs from the Phase 6 baseline (not from thin air), write an error-budget
policy, and wire multi-burn-rate alerts.
- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
of chaos flags, find the pathologies from telemetry alone using a top-down,
hypothesis-driven method, and write it up as an incident doc.
- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
OpenTelemetry on six concrete axes, using the platform's real second stack
(Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
- **10 — Package the delivery.** README, a rehearsed five-minute demo, the
written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`,
`CONCEPTS.md`), and an honest "what I'd do differently in production"
section.
## If you have to cut scope
Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not
cut Phase 8** — it's the only phase that proves you learned anything.