Add project documentation and Claude instructions
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
This commit is contained in:
Movses Aslanyan
@@ -0,0 +1,9 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.pytest_cache/
|
||||
.env
|
||||
*.egg-info/
|
||||
.DS_Store
|
||||
venv/
|
||||
.venv/
|
||||
@@ -0,0 +1,398 @@
|
||||
# CLAUDE.md — Ledgerline
|
||||
|
||||
Project instructions for Claude. Read this fully before acting.
|
||||
|
||||
---
|
||||
|
||||
## What this project is
|
||||
|
||||
**Ledgerline** is a deliberately imperfect seven-service Python/FastAPI system
|
||||
simulating a payments backend. The business logic is fake and does not matter.
|
||||
|
||||
The project exists as a **subject for learning observability**. The real
|
||||
deliverable is not working software — it is the user's understanding of
|
||||
OpenTelemetry instrumentation, the Grafana stack (Tempo / Loki / Prometheus /
|
||||
Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is
|
||||
assessed by a mentor, Babak.
|
||||
|
||||
The full plan lives in `docs/roadmap/`, split one file per phase so an agent
|
||||
working on a given phase doesn't have to load the other ten: `intro.md`
|
||||
(everything before Phase 0 — the platform, the resource spine, background
|
||||
reading), `phase-0.md` through `phase-10.md`, and `summary.md` (a one-page
|
||||
status/sequencing overview of all of them). Phase 0 is built test-first from
|
||||
two specs: `docs/phase-0a-test-suite-spec.md` (the tests) and
|
||||
`docs/phase-0-base-project-spec.md` (the system).
|
||||
|
||||
### The observability backend is external and pre-built
|
||||
|
||||
The Grafana stack is **not** built in this repo. It is deployed by a separate
|
||||
Ansible project, `observability-ansible`, which the user received from his mentor.
|
||||
It deploys two stacks on a target host via Docker Compose:
|
||||
|
||||
- **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with
|
||||
**Grafana Alloy** as the collector and node-exporter for host metrics.
|
||||
- **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the
|
||||
hands-on subject of Phase 9, not part of the main pipeline.
|
||||
|
||||
Five facts about that platform that change what correct code looks like here. Verify
|
||||
them against the live system rather than assuming — they were read from the configs
|
||||
at one point in time and the user may have since extended them.
|
||||
|
||||
1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
|
||||
`traces` and nothing else. OTLP **metrics and logs are accepted and silently
|
||||
discarded** — no error anywhere. This is deliberate: the platform's own diagram
|
||||
labels the OTLP input "spans + resource attrs" and draws app metrics as a
|
||||
separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
|
||||
metrics "should be arriving"; check Prometheus or the Alloy UI first.
|
||||
2. **App metrics reach Prometheus by direct scrape, not through Alloy.** Alloy has
|
||||
no `prometheus.scrape` component; `prometheus.yml.j2` scrapes the stack's own
|
||||
components plus whatever is listed in `prometheus_extra_scrape_jobs`. Note that
|
||||
the diagram draws this differently (app metrics routed via Alloy) — the diagram
|
||||
and the code have drifted, and the user knows. Don't "correct" one to the other
|
||||
unasked.
|
||||
3. **Span metrics and the service graph come from Tempo's metrics-generator**
|
||||
(`processors: [service-graphs, span-metrics, local-blocks]`, remote-writing to
|
||||
Prometheus with `send_exemplars: true`) — **not** from collector connectors.
|
||||
Never add `spanmetrics` / `servicegraph` to Alloy; it duplicates every series.
|
||||
The metric names differ from the ones in most blog posts — look them up in
|
||||
Prometheus before writing PromQL.
|
||||
4. **Log→trace correlation runs on a case-sensitive regex:**
|
||||
`trace[_-]?id["=: ]+([a-fA-F0-9]+)`. Logs must carry **lowercase `trace_id`**.
|
||||
OpenTelemetry's logging auto-instrumentation emits `otelTraceID`, which does not
|
||||
match. Exemplar labels must also be `trace_id`. The diagram names this outright:
|
||||
"Correlation key: trace_id / exemplars."
|
||||
5. **There is no dashboard provisioning in the platform** — datasources only.
|
||||
Dashboards-as-code is the user's deliverable and belongs in the Ansible role, so
|
||||
that `ansible-playbook` rebuilds them.
|
||||
|
||||
Ledgerline talks to this platform over the network. Endpoints come from env
|
||||
(`OTEL_EXPORTER_OTLP_ENDPOINT` etc.), never hardcoded.
|
||||
|
||||
### The single most important thing to understand
|
||||
|
||||
**The application is broken on purpose.** It contains thirteen deliberate
|
||||
pathologies — a missing index, a starved connection pool, an N+1 query, a
|
||||
blocking call inside an `async def`, a retry storm with no backoff, an unbounded
|
||||
cache, and more. They are documented in `docs/PATHOLOGIES.md` and toggled at
|
||||
runtime via Redis-backed chaos flags.
|
||||
|
||||
**Never fix them. Never suggest fixing them. Never "helpfully" clean them up
|
||||
while doing something else nearby.** Seeing `pool_size=2` or a `time.sleep()`
|
||||
in an async handler and correcting it destroys weeks of the user's work.
|
||||
If a pathology genuinely blocks something, say so and ask — don't act.
|
||||
|
||||
---
|
||||
|
||||
## Spoiler rules — hard constraints
|
||||
|
||||
These override helpfulness. Breaking any of them ruins the exercise.
|
||||
|
||||
1. **Do not read, summarise, quote, paraphrase, or hint at the contents of
|
||||
`docs/PATHOLOGIES.md`** unless the user explicitly says they have finished
|
||||
Phase 8 and wants to review it. Not even "let me check the docs to confirm."
|
||||
2. **Do not reveal which chaos flags are currently enabled** during Phase 8. Do
|
||||
not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and
|
||||
do not infer the set from the source and announce it. If the user asks you to
|
||||
check, remind them what phase they're in and let them insist — then comply.
|
||||
3. **Do not run `pytest -m pathology` during Phase 8.** The pathology suite has
|
||||
one named test per flag, so its pass/fail output is a complete answer key.
|
||||
Same for `make test-pathology` and `make test-slow`. Running the `contract`,
|
||||
`seed`, `baseline` or `guard` markers is fine.
|
||||
4. **Do not diagnose the system for the user during Phase 8.** That phase is the
|
||||
capstone: they find the bottlenecks from telemetry alone. Your job there is
|
||||
to teach *method* (how to read a trace gap, how to reason about saturation)
|
||||
and to operate tools they ask you to operate — not to name the culprit. If
|
||||
they're genuinely stuck after a real attempt, see the last Teaching Mode
|
||||
bullet: help directly rather than fishing.
|
||||
5. Outside Phase 8 these relax. Ask which phase they're in if it's unclear.
|
||||
|
||||
---
|
||||
|
||||
## Current phase
|
||||
|
||||
> **Phase: 0a** — writing the integration test suite.
|
||||
> *(The user updates this line. If it looks stale relative to the conversation,
|
||||
> ask before assuming.)*
|
||||
|
||||
Phase reference: **0a** test suite · **0b** implement to green · 1 concepts +
|
||||
first trace · 2 tracing · 3 logs · 4 metrics · 5 dashboards-as-code ·
|
||||
6 k6 + percentiles · 7 SLO/error budget · 8 bottleneck hunt ·
|
||||
9 Filebeat/Logstash/OTel judgment · 10 delivery.
|
||||
|
||||
---
|
||||
|
||||
## Teaching Mode
|
||||
|
||||
*Portable rule — copy this section into any project's `CLAUDE.md` as-is.*
|
||||
|
||||
When the user asks to be *taught* a piece of work rather than have it done
|
||||
for them (building a config file, writing a script, running a sequence of
|
||||
commands, standing up some infrastructure), switch out of the default
|
||||
"just complete the task" workflow and into this one instead. Stay in it
|
||||
for the rest of that work item unless the user opts out.
|
||||
|
||||
- **Never produce the finished artifact yourself.** Break it into small,
|
||||
sequential checkpoints instead of one big deliverable. Introduce one
|
||||
checkpoint at a time — what it needs to do and why — then stop and wait
|
||||
for the user's own attempt before moving to the next one.
|
||||
- **Teach the concept before asking for an attempt**, whenever a checkpoint
|
||||
depends on something the user hasn't demonstrated familiarity with yet.
|
||||
Ground the explanation in the actual thing being built, not in the
|
||||
abstract — use the user's own file/system as the running example.
|
||||
- **Verify their actual work, not their claim about it.** When they say
|
||||
"I did X" or "check this," go read the file / inspect the live state
|
||||
yourself before responding. Don't take it on faith.
|
||||
- **When something's wrong, give why + how + how to verify — never just
|
||||
the fix.** State the underlying mechanism that makes it wrong (not just
|
||||
"this is incorrect"), point at what needs to change conceptually so the
|
||||
user writes the correction themselves, and show them how to confirm the
|
||||
fix actually worked once they've made it.
|
||||
- **Prefer real verification over asserted theory, for you as much as for
|
||||
them.** When you're not fully certain how something will actually
|
||||
behave — or even when you are, if it's cheap to check — run it for real
|
||||
against the live system rather than describing what "should" happen. If
|
||||
a prediction turns out wrong once verified, say so plainly and correct
|
||||
course instead of quietly rewriting the story to fit.
|
||||
- **Use scratch/throwaway resources for exploratory experiments** that
|
||||
aren't part of the user's real deliverable (settling a "what actually
|
||||
happens if..." question), and clean them up afterward so they don't
|
||||
pollute the real project state. Ask before deleting anything that might
|
||||
be real work rather than obvious scratch output.
|
||||
- **Stop withholding once the user is genuinely stuck.** If they say "I
|
||||
don't know" or make a real attempt and can't get further, answer
|
||||
directly instead of continuing to fish for guesses — the goal is
|
||||
understanding, not making them struggle for its own sake.
|
||||
- **When the user proposes their own explanation for confusing behavior,
|
||||
test it rather than defaulting to your own theory.** Treat it as a real
|
||||
hypothesis worth verifying, not something to talk them out of.
|
||||
|
||||
---
|
||||
|
||||
## Teaching Mode scope in this project
|
||||
|
||||
Teaching Mode is the **default** here. But not everything in this repo is
|
||||
learning material, and treating scaffolding as a lesson wastes the user's time.
|
||||
|
||||
Decide by what's being touched:
|
||||
|
||||
| Path / concern | Mode |
|
||||
|---|---|
|
||||
| `services/**` business logic, `scripts/seed.py`, `db/schema.sql`, frontend, Compose plumbing, Makefile | **Just build it.** This is scaffolding. The user has stated he doesn't care about the microservices themselves. |
|
||||
| `tests/**` | **Just build it** during Phase 0a, from the test spec. Afterwards it is a **frozen contract** — see the rules below. |
|
||||
| `common/telemetry.py`, any OTel SDK code inside services, span/metric/log instrumentation | **Teaching Mode.** |
|
||||
| `observability/**` — Collector config, Grafana provisioning, datasource cross-links, dashboards-as-code, alert rules | **Teaching Mode.** |
|
||||
| `loadtest/**` — k6 scripts, executors, thresholds | **Teaching Mode.** |
|
||||
| `docs/SLO.md`, `docs/INVESTIGATION.md`, `docs/PIPELINE-CHOICES.md`, `docs/CONCEPTS.md` | **Teaching Mode** — these are the user's own reasoning, written in his own words. Never draft them for him. Review and challenge instead. |
|
||||
| PromQL / LogQL / TraceQL queries | **Teaching Mode.** Query languages are a skill, not a fact. |
|
||||
|
||||
**Phase 0 is explicitly exempt**, in both halves. The base project was delegated
|
||||
to coding agents by design: **0a** writes the tests, **0b** implements against
|
||||
them. Build both from their specs without teaching. Phases 1–10 are the learning
|
||||
work.
|
||||
|
||||
If the user says "just do it," "give me the code," or "skip the socratic
|
||||
thing" — comply immediately and without negotiation for that work item.
|
||||
|
||||
### The test suite is a contract
|
||||
|
||||
Phase 0 is built test-first by two separate agents, and the whole value of that
|
||||
split collapses if the tests drift toward the implementation.
|
||||
|
||||
- **During 0a:** write only `tests/`. Do not create stubs, scaffolding, or
|
||||
placeholder modules under `services/`, `common/`, `db/` or `scripts/` — not
|
||||
even to make imports resolve. Tests talk to the system over HTTP, Postgres,
|
||||
Redis and the Docker CLI, never by importing it.
|
||||
- **During 0b and afterwards:** `tests/` is **read-only**. Never edit, delete,
|
||||
skip, `xfail`, or loosen a test, a threshold, a marker, or a fixture to make
|
||||
code pass. If a test looks wrong or impossible, **stop and say so** — the user
|
||||
decides, not you.
|
||||
- **Never fake a pathology to satisfy a timing assertion.** A `sleep()` tuned to
|
||||
clear a latency ratio is a failure even when the suite goes green. Implement
|
||||
the mechanism described in the Phase 0 spec §7. Several assertions (query
|
||||
counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception
|
||||
paths) are deliberately built so faking is harder than implementing; treat that
|
||||
as the intent for all of them.
|
||||
- From Phase 3 onward the log-line-counting assertions in the pathology suite
|
||||
*should* be rewritten against structured logs and then span/metric data. That
|
||||
is a planned migration the user drives — propose it, don't do it unasked, and
|
||||
it does not license loosening anything else.
|
||||
|
||||
---
|
||||
|
||||
## Calibrating explanations to this user
|
||||
|
||||
Senior Python engineer, 7+ years. Django, FastAPI, PostgreSQL, SQLAlchemy,
|
||||
Celery, Docker, AWS/GCP, Terraform, Ansible, Kubernetes. Runs his own
|
||||
single-node cluster.
|
||||
|
||||
- **Don't explain:** Python, async/await, FastAPI, Pydantic, SQLAlchemy, SQL,
|
||||
`EXPLAIN ANALYZE`, Docker, Compose, HTTP, REST, git, Linux.
|
||||
- **Do explain:** the OpenTelemetry data model and API/SDK/Collector split,
|
||||
context propagation mechanics, Prometheus's series/cardinality model, PromQL,
|
||||
LogQL, TraceQL, Loki's label vs body distinction, span metrics and service
|
||||
graph connectors, exemplars, k6 executors and coordinated omission, SLI/SLO
|
||||
arithmetic and burn rates.
|
||||
- Connect new concepts to things he already knows where it genuinely helps —
|
||||
`contextvars` for in-process trace context, connection-pool queueing for
|
||||
saturation, `EXPLAIN ANALYZE` for what a DB span can't show you.
|
||||
- He learns from professional reference implementations. When a pattern is
|
||||
non-obvious, point him at the relevant file in the OpenTelemetry Demo
|
||||
(`open-telemetry/opentelemetry-demo`) rather than inventing an example.
|
||||
- He extracts study notes from video subtitles. When recommending video, note
|
||||
whether captions exist.
|
||||
|
||||
---
|
||||
|
||||
## Verification — check the live system, don't assert
|
||||
|
||||
Teaching Mode's "verify their actual work" and "prefer real verification"
|
||||
bullets are not optional here. Concrete ways to do it in this stack (confirm
|
||||
ports against `docker-compose.yml` — these are the intended defaults):
|
||||
|
||||
```bash
|
||||
docker compose ps # what's actually running
|
||||
make smoke # end-to-end sanity
|
||||
docker compose logs -f gateway # actual log shape, not the intended one
|
||||
|
||||
make test-contract # fast functional suite (< 60s)
|
||||
make test-baseline # is the clean system actually healthy
|
||||
pytest tests/ -m guard -q # "no observability yet" — delete at Phase 2
|
||||
git diff --stat tests/ # did anything touch the contract? should be empty
|
||||
```
|
||||
|
||||
The pathology suite (`make test-pathology`, `make test-slow`) is slow and is an
|
||||
answer key — see spoiler rule 3 before running it.
|
||||
|
||||
**Traces (Tempo, :3200)** — search by service and duration rather than trusting
|
||||
that a span exists:
|
||||
```bash
|
||||
curl -s 'http://localhost:3200/api/search?tags=service.name%3Dledger&limit=5'
|
||||
curl -s "http://localhost:3200/api/traces/<trace_id>" | jq '.batches | length'
|
||||
```
|
||||
|
||||
**Metrics (Prometheus, :9090)** — and this is the cardinality check worth
|
||||
running often:
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/query?query=count({__name__=~"ledgerline.*"})'
|
||||
curl -s 'http://localhost:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'
|
||||
```
|
||||
|
||||
**Logs (Loki, :3100)** — confirm `trace_id` is actually in the payload, not just
|
||||
in the code that was supposed to put it there:
|
||||
```bash
|
||||
curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
|
||||
--data-urlencode 'query={service_name="gateway"} | json | trace_id != ""' \
|
||||
--data-urlencode 'limit=5'
|
||||
```
|
||||
|
||||
**Alloy (:12345)** — the component graph is the fastest way to see what the pipeline
|
||||
actually does, including which signal outputs are wired. Check it before theorising
|
||||
about missing data.
|
||||
|
||||
**Metric names** — never assume; the Tempo-generated names differ from blog posts:
|
||||
```bash
|
||||
curl -s 'http://<obs-host>:9090/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))'
|
||||
```
|
||||
|
||||
**Grafana (:3000)** — verify provisioning actually loaded rather than reading
|
||||
the YAML and assuming:
|
||||
```bash
|
||||
curl -s -u admin:admin 'http://localhost:3000/api/datasources' | jq '.[].name'
|
||||
curl -s -u admin:admin 'http://localhost:3000/api/search?type=dash-db' | jq '.[].title'
|
||||
```
|
||||
|
||||
Rules of thumb:
|
||||
|
||||
- After the user says "I added the span" / "the exporter is configured" — go
|
||||
look at the file *and* query the backend. Code present ≠ data arriving. The
|
||||
gap between those two is where most of the real learning is.
|
||||
- Before predicting how the Collector, a sampler, or a PromQL query will behave:
|
||||
if it's cheap to run, run it. Batching, sampling and cardinality behaviour are
|
||||
all easy to get wrong from memory.
|
||||
- If a prediction turns out wrong, say so plainly and correct course.
|
||||
|
||||
---
|
||||
|
||||
## Scratch resources
|
||||
|
||||
For "what actually happens if…" experiments that aren't part of the deliverable:
|
||||
|
||||
- Scratch dashboards → prefix `scratch-`, uid prefix `scratch-`.
|
||||
- Scratch metrics → prefix `scratch_`.
|
||||
- Scratch Compose services → `observability/scratch.compose.yml`, never the main
|
||||
file.
|
||||
- Scratch scripts → `/tmp/`, not the repo.
|
||||
|
||||
Clean up afterwards. **Never delete** anything under `observability/`,
|
||||
`loadtest/`, `docs/`, or any dashboard without a `scratch-` prefix without
|
||||
asking first — a hand-tuned dashboard or a half-written investigation doc is
|
||||
real work even if it looks messy.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns specific to this project
|
||||
|
||||
Things a well-meaning assistant does wrong here:
|
||||
|
||||
- **Fixing pathologies.** Adding an index, raising `pool_size`, wrapping the
|
||||
blocking call in `to_thread`, adding backoff to the retry loop. Never.
|
||||
- **Adding observability during Phase 0.** The base app ships with plain-text
|
||||
stdlib logging and a weak manual `X-Request-Id`. That's the "before" state
|
||||
the whole project measures against. `tests/test_no_observability.py` enforces
|
||||
it — if that test fails, something crept in; don't delete the test to fix it.
|
||||
- **Editing a test to make code pass.** Including "just relaxing the threshold,"
|
||||
adding `@pytest.mark.skip`, or widening an assertion because a run was noisy.
|
||||
Report it instead.
|
||||
- **Satisfying a pathology test with a `sleep()`** rather than the real
|
||||
mechanism.
|
||||
- **Importing application code into `tests/`.** The suite is black-box on
|
||||
purpose; an import couples it to implementation and the contract stops meaning
|
||||
anything.
|
||||
- **Writing `common/telemetry.py` for him** in Phase 2. That file is the single
|
||||
most valuable thing he writes. Checkpoint it: Resource → TracerProvider →
|
||||
BatchSpanProcessor → exporter → propagator, one at a time.
|
||||
- **Creating or editing dashboards through the Grafana UI or HTTP API** as a
|
||||
deliverable. Everything must come from provisioning files or the Foundation
|
||||
SDK generator, because "destroy and recreate in seconds" is an explicit
|
||||
requirement from Babak. Read-only API calls for *verification* are fine.
|
||||
- **Writing OpenTelemetry Collector YAML.** The platform uses **Alloy**, whose
|
||||
config is component-based (`otelcol.receiver.otlp`, `otelcol.exporter.otlp`,
|
||||
`loki.source.docker`, `prometheus.remote_write`), not YAML receiver/processor/
|
||||
exporter pipelines. A correct-looking Collector config is a wrong answer here.
|
||||
- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
|
||||
already does this.
|
||||
- **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in
|
||||
Prometheus / Loki before claiming anything works.
|
||||
- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
|
||||
Loki derived field with no visible error.
|
||||
- **`avg(p99)` or averaging any quantile** across instances or time buckets.
|
||||
Aggregate the histogram buckets, then compute the quantile.
|
||||
- **High-cardinality metric labels** — `account_id`, `transaction_id`,
|
||||
`merchant_name`, raw paths. These belong on spans. (There is one *deliberate*
|
||||
cardinality-explosion exercise in Phase 4; that one is intentional and gets
|
||||
reverted.)
|
||||
- **`constant-vus` for load profiles** where an open model is correct.
|
||||
Closed-model executors hide latency through coordinated omission.
|
||||
- **Putting `trace_id` in a Loki label** instead of the log body + derived field.
|
||||
- **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a
|
||||
Phase 9 judgment exercise and a small side pipeline only — not the
|
||||
architecture.
|
||||
- Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md`
|
||||
for him. Those are his reasoning. Review, challenge, poke holes — don't write.
|
||||
|
||||
---
|
||||
|
||||
## Conventions
|
||||
|
||||
- Python 3.12, FastAPI, SQLAlchemy 2.0 async + asyncpg, `httpx` async,
|
||||
`redis-py` asyncio, Pydantic v2. No Celery, no RabbitMQ — Redis Streams,
|
||||
deliberately, because it has no automatic context propagation.
|
||||
- One Docker image for all services; `CMD` differs per service.
|
||||
- Semantic conventions for standard attributes (`http.request.method`,
|
||||
`db.system`, `server.address`). Namespace custom ones as `ledgerline.*`.
|
||||
- `service.name` matches the directory name under `services/`.
|
||||
- Duplication across services is fine. Don't refactor toward a shared framework.
|
||||
- Docker Compose only, locally, through Phase 9. Kubernetes is a later
|
||||
follow-up — porting instrumentation and debugging a cluster at the same time
|
||||
is how this stalls.
|
||||
- Never save a Grafana dashboard from the UI.
|
||||
@@ -0,0 +1,371 @@
|
||||
# Ledgerline
|
||||
|
||||
A deliberately imperfect seven-service payments backend, built as a subject for
|
||||
learning observability with OpenTelemetry and the Grafana stack.
|
||||
|
||||
> ### ⚠️ This system is broken on purpose
|
||||
>
|
||||
> Ledgerline ships with thirteen intentional performance and reliability
|
||||
> pathologies: a missing index, a starved connection pool, an N+1 query, a
|
||||
> blocking call inside an `async def`, a retry storm with no backoff, an
|
||||
> unbounded cache, and more. They are toggled at runtime and documented in
|
||||
> [`docs/PATHOLOGIES.md`](docs/PATHOLOGIES.md).
|
||||
>
|
||||
> **Do not fix them.** They are the entire point. If you are here to contribute,
|
||||
> read [`CLAUDE.md`](CLAUDE.md) first.
|
||||
|
||||
The business logic is fake and does not need to be correct, secure, or
|
||||
defensible. What it needs to be is *structurally* realistic: real network hops,
|
||||
real SQL, real async message passing, real contention, and real latency
|
||||
distributions — so that instrumenting it teaches something true.
|
||||
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
|
||||
Requires Docker and Docker Compose. Nothing else.
|
||||
|
||||
```bash
|
||||
git clone <this-repo> && cd ledgerline
|
||||
cp .env.example .env
|
||||
make up # start postgres, redis, and the seven services
|
||||
make seed # schema + 250k transactions across 800 accounts (~60s)
|
||||
make smoke # end-to-end sanity check
|
||||
make test-contract # fast functional suite (~60s)
|
||||
```
|
||||
|
||||
Then open **http://localhost:8000**.
|
||||
|
||||
The observability backend lives in a separate Ansible repo and is deployed to its
|
||||
own host. From Phase 1 onward:
|
||||
|
||||
```bash
|
||||
# in observability-ansible/
|
||||
ansible-playbook playbooks/grafana-stack.yml
|
||||
```
|
||||
|
||||
Then point Ledgerline at it via `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env`.
|
||||
|
||||
Reset everything, including volumes:
|
||||
|
||||
```bash
|
||||
make reset
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
B[browser] --> GW[gateway :8000]
|
||||
GW --> LD[ledger :8001]
|
||||
GW --> FR[fraud :8002]
|
||||
FR --> SA[sanctions :8003]
|
||||
GW --> IN[insights :8004]
|
||||
GW -- XADD --> ST[(redis stream)]
|
||||
ST --> NO[notifier :8005]
|
||||
ST --> AU[audit worker]
|
||||
LD --> PG[(postgres)]
|
||||
SA --> PG
|
||||
IN --> PG
|
||||
NO --> PG
|
||||
AU --> PG
|
||||
IN --> RD[(redis cache)]
|
||||
```
|
||||
|
||||
| Service | Port | Role | Stores |
|
||||
|---|---|---|---|
|
||||
| `gateway` | 8000 | Public API, serves the frontend, orchestrates everything | Redis (flags) |
|
||||
| `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
|
||||
| `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
|
||||
| `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow vendor" | Postgres (read) |
|
||||
| `insights` | 8004 | 120-day analytics over transaction history | Postgres + Redis |
|
||||
| `notifier` | 8005 | HTTP API **and** a Redis Stream consumer | Postgres |
|
||||
| `audit` | — | Worker only; second consumer group on the same stream | Postgres |
|
||||
|
||||
Every HTTP service exposes `GET /healthz` (liveness) and `GET /readyz`
|
||||
(dependency check).
|
||||
|
||||
**Redis Streams is used deliberately** rather than Celery or RabbitMQ: it has no
|
||||
automatic trace-context propagation, so Phase 2 has to implement it by hand.
|
||||
That's the most valuable exercise in the project. Don't swap it out.
|
||||
|
||||
### The primary request flow
|
||||
|
||||
`POST /api/transactions` produces a trace roughly four levels deep with one
|
||||
async fork:
|
||||
|
||||
```
|
||||
gateway
|
||||
├─ ledger create transaction (pending, row lock on account)
|
||||
├─ fraud score
|
||||
│ └─ sanctions watchlist lookup
|
||||
├─ ledger settle or reject
|
||||
├─ redis XADD stream:transactions ──┬─ notifier (consumer group)
|
||||
│ └─ audit (consumer group)
|
||||
└─ insights cached account summary
|
||||
```
|
||||
|
||||
The second flow, `GET /api/accounts/{id}/insights`, is a read-heavy aggregation
|
||||
and behaves very differently under load.
|
||||
|
||||
---
|
||||
|
||||
## Chaos flags
|
||||
|
||||
Pathologies are controlled at runtime via a Redis hash (`chaos:flags`), read on
|
||||
every request — **no restart needed**. Toggle them from the frontend's chaos
|
||||
panel, the API, or the Makefile.
|
||||
|
||||
```bash
|
||||
make chaos-clean # all off — the baseline
|
||||
make chaos-light # a mild, realistic amount of pain
|
||||
make chaos-nightmare # all thirteen on
|
||||
make chaos-random # 3–6 flags, seed never logged
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -s localhost:8000/api/admin/chaos | jq
|
||||
curl -sX POST localhost:8000/api/admin/chaos \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"flag":"slow_sanctions","enabled":true}'
|
||||
```
|
||||
|
||||
Flag names: `slow_sanctions`, `sanctions_errors`, `n_plus_one`,
|
||||
`pool_starvation`, `row_lock_contention`, `blocking_event_loop`, `retry_storm`,
|
||||
`no_batching`, `unbounded_cache`, `slow_consumer`, `cold_cache`, `poison_data`,
|
||||
`chatty_health`.
|
||||
|
||||
> **During Phase 8, don't open the chaos panel and don't call
|
||||
> `GET /api/admin/chaos`.** `make chaos-random` picks a subset and never logs the
|
||||
> seed; the point is to find the pathologies from telemetry alone. The panel
|
||||
> will happily tell you the answer, which is the one thing you don't want.
|
||||
|
||||
---
|
||||
|
||||
## Make targets
|
||||
|
||||
| Target | Effect |
|
||||
|---|---|
|
||||
| `up` / `down` | Start / stop the application stack |
|
||||
| `logs` | `docker compose logs -f` |
|
||||
| `seed` | Apply schema and seed 250k transactions |
|
||||
| `reset` | `down -v` → `up` → `seed` |
|
||||
| `smoke` | One transaction + one insights call, assert 200s |
|
||||
| `test` | Full suite except the slow markers |
|
||||
| `test-contract` | Functional + seed + guard assertions, < 60 s |
|
||||
| `test-baseline` | Is the clean system genuinely healthy |
|
||||
| `test-pathology` | One differential test per pathology — ⚠️ answer key, see below |
|
||||
| `test-slow` | The soak-ish pathology tests |
|
||||
| `chaos-clean` / `-light` / `-nightmare` / `-random` | Set the chaos profile |
|
||||
| `fix-index` / `drop-index` | Add or remove the deliberately missing indexes |
|
||||
| `dashboards` | Regenerate dashboard JSON via the Grafana Foundation SDK — *Phase 5+* |
|
||||
| `load-smoke` / `load-average` / `load-stress` / `load-soak` | k6 profiles — *Phase 6+* |
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Phase 0 was built **test-first, by two separate agents**: one wrote `tests/`
|
||||
from `docs/phase-0a-test-suite-spec.md` without seeing any implementation, the
|
||||
second implemented the services until the suite went green without being allowed
|
||||
to touch a test.
|
||||
|
||||
The suite is **black-box**. It imports nothing from the application and talks to
|
||||
the system only over HTTP, Postgres, Redis and the Docker CLI. It asserts goals,
|
||||
not mechanics — there are no references to function names, module paths, or SQL
|
||||
strings anywhere in it.
|
||||
|
||||
Two halves:
|
||||
|
||||
- **Working state.** A transaction really traverses all seven services (stop
|
||||
`sanctions` and transactions must fail — that's the test that catches a
|
||||
collapsed monolith); async work really happens off the request path via two
|
||||
independent consumer groups; the seed data is really 250k rows, really skewed
|
||||
toward five hot accounts, and really contains all five varieties of poison row.
|
||||
- **Broken state.** One differential test per pathology, asserting the system
|
||||
measurably degrades when a flag is on versus off. Assertions favour signals
|
||||
that are hard to fake: `pg_stat_statements` query counts for the N+1,
|
||||
`EXPLAIN` plan node types for the missing index, Redis consumer-group lag for
|
||||
the slow consumer, container RSS for the memory leak, throughput plateaus for
|
||||
pool starvation. A `sleep()` won't satisfy those.
|
||||
|
||||
```bash
|
||||
make up && make seed # prerequisite — the suite runs against a live stack
|
||||
make test-contract # fast
|
||||
make test # everything except slow
|
||||
```
|
||||
|
||||
> ⚠️ **`make test-pathology` is a spoiler.** It has one named test per flag, so
|
||||
> its output is a complete answer key. Don't run it during the Phase 8
|
||||
> bottleneck hunt.
|
||||
|
||||
`tests/test_no_observability.py` guards the Phase 0 "before" state — no OTel, no
|
||||
`prometheus_client`, no structured logging, no `/metrics`. **Delete it at the
|
||||
start of Phase 2**; it has done its job by then.
|
||||
|
||||
Some pathology assertions currently count log lines, because plain text is the
|
||||
only cross-service observability surface that exists at Phase 0. That is the
|
||||
deficiency this project sets out to fix — those assertions get rewritten against
|
||||
structured logs in Phase 3 and span data in Phase 4, and become both simpler and
|
||||
more robust. See `tests/README.md`.
|
||||
|
||||
---
|
||||
|
||||
## Ports
|
||||
|
||||
| | Port | |
|
||||
|---|---|---|
|
||||
| Frontend / gateway | 8000 | |
|
||||
| `ledger` … `notifier` | 8001–8005 | |
|
||||
| Postgres | 5432 | `ledgerline` / `ledgerline` |
|
||||
| Redis | 6379 | |
|
||||
|
||||
On the observability host (deployed by `observability-ansible`):
|
||||
|
||||
| | Port | |
|
||||
|---|---|---|
|
||||
| Grafana | 3000 | credentials from `group_vars/all.yml` |
|
||||
| Prometheus | 9090 | remote-write receiver enabled |
|
||||
| Alertmanager | 9093 | |
|
||||
| Loki | 3100 | |
|
||||
| Tempo | 3200 | |
|
||||
| Alloy UI | 12345 | component graph — check here first when data is missing |
|
||||
| OTLP (Alloy) | 4317 / 4318 | gRPC / HTTP |
|
||||
| Node Exporter | 9100 | |
|
||||
|
||||
---
|
||||
|
||||
## Observability stack
|
||||
|
||||
The backend is **not** part of this repo. It is deployed by a separate Ansible
|
||||
project, `observability-ansible`, onto a target host:
|
||||
|
||||
```
|
||||
Ledgerline /metrics ──────scrape──────────────┐
|
||||
container stdout ──docker.sock──▶ Alloy ──▶ Loki
|
||||
Ledgerline OTLP :4317/4318 ─────▶ Alloy ──▶ Tempo
|
||||
│ span metrics + exemplars
|
||||
▼ (remote_write)
|
||||
Prometheus ◀────────┘
|
||||
│
|
||||
Alertmanager ──▶ Receivers
|
||||
│
|
||||
Grafana ◀── PromQL / LogQL / TraceQL ──▶ all of the above
|
||||
Correlation key: trace_id / exemplars
|
||||
```
|
||||
|
||||
RED metrics and the service graph are generated by **Tempo's metrics-generator**,
|
||||
not by collector connectors — do not add `spanmetrics`/`servicegraph` to Alloy.
|
||||
|
||||
Three constraints inherited from that platform, worth knowing before you debug
|
||||
anything:
|
||||
|
||||
- **Alloy's OTLP receiver forwards traces only.** OTLP metrics and logs are
|
||||
accepted and silently dropped — by design; the platform's diagram labels that
|
||||
input "spans + resource attrs". Metrics reach Prometheus by **direct scrape** of
|
||||
each service's `/metrics` endpoint, registered in `prometheus_extra_scrape_jobs`.
|
||||
- **Log→trace correlation matches `trace[_-]?id["=: ]+([a-fA-F0-9]+)`,
|
||||
case-sensitively.** Log lines must carry lowercase `trace_id`. OTel's logging
|
||||
auto-instrumentation emits `otelTraceID` and will not match.
|
||||
- **The platform provisions datasources only.** Every dashboard here is generated
|
||||
from code with the Grafana Foundation SDK and shipped through the Ansible role,
|
||||
so `ansible-playbook` rebuilds them from scratch. **Never save a dashboard from
|
||||
the Grafana UI** — it won't survive the next run.
|
||||
|
||||
Set `OTEL_EXPORTER_OTLP_ENDPOINT` (and friends) in `.env` to point at the Alloy
|
||||
host. Nothing observability-related is hardcoded.
|
||||
|
||||
---
|
||||
|
||||
## The five-minute demo
|
||||
|
||||
Rehearse this before showing anyone.
|
||||
|
||||
1. Delete every dashboard in the Grafana UI, then re-run
|
||||
`ansible-playbook playbooks/grafana-stack.yml` — platform and dashboards both
|
||||
rebuilt from code, one command.
|
||||
2. Submit one transaction from the frontend. Follow that single request through
|
||||
Grafana: RED dashboard → latency spike → exemplar → trace → the slow span →
|
||||
that span's logs → back to the trace. One request, three signals, no
|
||||
searching.
|
||||
3. `make chaos-nightmare && make load-stress` — watch p95 diverge from p50, the
|
||||
service map turn red at a specific node, the error budget burn, and the
|
||||
fast-burn alert fire.
|
||||
4. Walk through [`docs/INVESTIGATION.md`](docs/INVESTIGATION.md) — how the
|
||||
pathologies were found from telemetry alone.
|
||||
|
||||
---
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
├─ CLAUDE.md # instructions for AI assistants — read before contributing
|
||||
├─ Makefile
|
||||
├─ docker-compose.yml
|
||||
├─ Dockerfile # one image, all Python services
|
||||
├─ db/
|
||||
│ ├─ schema.sql
|
||||
│ ├─ optional_indexes.sql # the deliberately-omitted indexes
|
||||
│ └─ seed_manifest.json # written by the seeder; the tests' source of truth
|
||||
├─ scripts/ # seed.py, smoke.py
|
||||
├─ common/ # settings, db, redis, http, logging, telemetry
|
||||
├─ services/ # gateway ledger fraud sanctions insights notifier audit
|
||||
│ └─ */chaos.py # each service's pathologies, clearly marked
|
||||
├─ tests/ # black-box integration suite — a frozen contract
|
||||
│ ├─ harness/ # load driver, pg_stat helpers, stream + container probes
|
||||
│ └─ README.md # methodology, markers, known-flaky list
|
||||
├─ observability/ # dashboard generator (Foundation SDK); output ships via the Ansible role
|
||||
├─ loadtest/ # k6 scripts
|
||||
└─ docs/
|
||||
├─ roadmap/ # the ten-phase learning plan, one file per phase
|
||||
│ ├─ intro.md # platform background, resource spine (read first)
|
||||
│ ├─ phase-0.md … phase-10.md # one phase per file — status, message, concepts, build
|
||||
│ └─ summary.md # one-page status/sequencing overview of all phases
|
||||
├─ phase-0a-test-suite-spec.md # spec the test suite was written from
|
||||
├─ phase-0-base-project-spec.md # spec the base app was built from
|
||||
├─ PATHOLOGIES.md # ⚠️ spoilers — the answer key
|
||||
├─ SLO.md # SLIs, SLOs, SLA, RTO/RPO, error budget policy
|
||||
├─ CONCEPTS.md # P95/P99, SLO vs SLA vs RTO vs RPO, in my own words
|
||||
├─ INVESTIGATION.md # the Phase 8 bottleneck hunt, written as an incident doc
|
||||
└─ PIPELINE-CHOICES.md # Filebeat vs Logstash vs OpenTelemetry — when and why
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
- [ ] **Phase 0a** — integration test suite, written first, red against nothing
|
||||
- [ ] **Phase 0b** — base project implemented until the suite is green
|
||||
- [ ] **Phase 1** — concepts, first trace end to end
|
||||
- [ ] **Phase 2** — distributed tracing across all seven services
|
||||
- [ ] **Phase 3** — structured logs, correlated with traces
|
||||
- [ ] **Phase 4** — metrics, RED, service map, exemplars
|
||||
- [ ] **Phase 5** — dashboards and alerts as code
|
||||
- [ ] **Phase 6** — k6 load testing, percentiles
|
||||
- [ ] **Phase 7** — SLIs, SLOs, error budgets, RTO/RPO
|
||||
- [ ] **Phase 8** — bottleneck hunt
|
||||
- [ ] **Phase 9** — Filebeat / Logstash / OpenTelemetry judgment
|
||||
- [ ] **Phase 10** — delivery
|
||||
|
||||
Full plan: [`docs/roadmap/`](docs/roadmap/) — start with
|
||||
[`intro.md`](docs/roadmap/intro.md), then the phase you're on, or
|
||||
[`summary.md`](docs/roadmap/summary.md) for the one-page overview.
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
No authentication, no authorization, no input hardening, no unit tests, no
|
||||
Kubernetes, no production-grade storage. Ledgerline runs on a laptop under
|
||||
Docker Compose and is never exposed to a network you don't control.
|
||||
|
||||
The integration suite is deliberately the *only* test layer: it exists to pin
|
||||
down the system's observable behaviour — working and broken — not to verify the
|
||||
fake business logic.
|
||||
|
||||
Things deliberately left for a later iteration, and worth naming when discussing
|
||||
the project: tail-based sampling instead of 100%, Mimir instead of raw
|
||||
Prometheus for retention, Alloy as a DaemonSet, log volume budgeting, PII
|
||||
redaction in the Collector, and multi-tenancy.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Pathology catalogue
|
||||
|
||||
Thirteen deliberate pathologies, toggled at runtime via the `chaos:flags` Redis
|
||||
hash (never a restart). Each lives in a `chaos.py` module inside its owning
|
||||
service. All flags default off.
|
||||
|
||||
> Spoiler note (see root `CLAUDE.md`): this file is a complete answer key for
|
||||
> the Phase 8 bottleneck hunt. Don't read it, and don't have it read to you,
|
||||
> until that phase is done.
|
||||
|
||||
---
|
||||
|
||||
## slow_sanctions (`sanctions`)
|
||||
|
||||
**Mechanism:** `services/sanctions/chaos.py`. On every `/internal/lookup`
|
||||
call, sleeps for a log-normal delay (median ≈ 40ms, p99 ≈ 1.8s). 1.5% of
|
||||
calls instead sleep a flat 5s -- longer than the 3s downstream timeout every
|
||||
caller uses (`common/http.py`), so these surface as timeouts one hop up, in
|
||||
`fraud`.
|
||||
|
||||
**Symptom:** heavy-tailed latency on `POST /api/transactions`; most requests
|
||||
look normal, a fat p95/p99 tail; occasional 504s from `fraud`/`gateway` with
|
||||
no corresponding 5xx logged in `sanctions` itself (it never errors -- it's
|
||||
just slow).
|
||||
|
||||
## sanctions_errors (`sanctions`)
|
||||
|
||||
**Mechanism:** `services/sanctions/chaos.py`. 3% of `/internal/lookup` calls
|
||||
return HTTP 503 outright, chosen independently per call via `random.random()`.
|
||||
|
||||
**Symptom:** a small, steady rate of 502/504 responses from `gateway`, fully
|
||||
uncorrelated with load or account. `sanctions` logs a "lookup" line for every
|
||||
attempt regardless, so its request count won't match its success count.
|
||||
|
||||
## n_plus_one (`insights`)
|
||||
|
||||
**Mechanism:** `services/insights/analytics.py::_n_plus_one_summary`. Instead
|
||||
of one `LEFT JOIN` against `merchant_categories`, fetches up to 500 raw
|
||||
transaction rows and issues one additional `SELECT ... FROM merchant_categories
|
||||
WHERE id = :id` per row.
|
||||
|
||||
**Symptom:** `GET /api/accounts/{id}/insights` for a high-volume (hot) account
|
||||
issues 50-500+ SQL statements instead of 1-2, and its latency grows with the
|
||||
account's transaction count rather than staying flat.
|
||||
|
||||
## pool_starvation (`ledger`)
|
||||
|
||||
**Mechanism:** `services/ledger/db.py`. Two engines are built at startup: a
|
||||
normal one (`pool_size=20, max_overflow=10`) and a starved one (`pool_size=2,
|
||||
max_overflow=0, pool_timeout=10`). Every `ledger` request picks which
|
||||
already-running pool to borrow a connection from based on the current flag --
|
||||
this is what makes the toggle live without a restart, since SQLAlchemy pools
|
||||
can't be resized in place.
|
||||
|
||||
**Symptom:** throughput plateaus as concurrency increases past ~2 in-flight
|
||||
requests; latency and queueing rise sharply under concurrent load while a
|
||||
single request in isolation still looks fine.
|
||||
|
||||
## row_lock_contention (`ledger`)
|
||||
|
||||
**Mechanism:** `services/ledger/repository.py::settle_transaction`. Only for
|
||||
accounts with `is_hot = true`: takes `SELECT balance_minor FROM accounts ...
|
||||
FOR UPDATE` and holds the transaction open for an extra 150ms before
|
||||
committing the balance update.
|
||||
|
||||
**Symptom:** concurrent transactions against a hot account show p95 latency
|
||||
several times higher than the same concurrency spread across many cold
|
||||
accounts; cold-account traffic is unaffected.
|
||||
|
||||
## blocking_event_loop (`fraud`)
|
||||
|
||||
**Mechanism:** `services/fraud/chaos.py::synchronous_cpu_work`. Runs ~120ms of
|
||||
synchronous SHA-256 hashing directly inside the `async def` handler for
|
||||
`/internal/score` -- no `run_in_executor`/`to_thread`.
|
||||
|
||||
**Symptom:** concurrent requests to `fraud` (and anything waiting on it)
|
||||
serialise: wall-clock time for N concurrent requests approaches N × single-request
|
||||
time instead of staying close to single-request time, even though nothing is
|
||||
CPU-bound at the infrastructure level.
|
||||
|
||||
## retry_storm (`gateway`)
|
||||
|
||||
**Mechanism:** `services/gateway/orchestrator.py::_score_with_retry`. On any
|
||||
non-2xx from `fraud`, retries immediately up to 3 more times -- no backoff,
|
||||
no jitter, no circuit breaker.
|
||||
|
||||
**Symptom:** when `sanctions` is also failing or slow, the number of
|
||||
`sanctions` "lookup" log lines per transaction multiplies (up to 4x), and
|
||||
downstream load spikes exactly when the downstream is already unhealthy --
|
||||
the opposite of what you want.
|
||||
|
||||
## no_batching (`gateway`)
|
||||
|
||||
**Mechanism:** `services/gateway/orchestrator.py::create_transaction`. Splits
|
||||
the transaction into 1-5 synthetic "line items" and calls `fraud` once per
|
||||
item, sequentially, instead of once for the whole transaction.
|
||||
|
||||
**Symptom:** `sanctions` lookups per transaction multiply (roughly 2-5x) and
|
||||
end-to-end latency for `POST /api/transactions` grows proportionally, even
|
||||
though sanctions/fraud themselves are healthy.
|
||||
|
||||
## unbounded_cache (`insights`)
|
||||
|
||||
**Mechanism:** `services/insights/chaos.py` / `app.py`. Instead of the normal
|
||||
Redis cache (keyed by account + current chaos-flag fingerprint, TTL below),
|
||||
memoizes summaries in a module-level dict keyed by `(account_id, fingerprint,
|
||||
time.time_ns())` -- a key that is different on every single call. It never
|
||||
hits, and it never evicts.
|
||||
|
||||
**Symptom:** `insights`' resident memory grows monotonically under repeated
|
||||
identical-looking requests and never plateaus or drops; cache hit rate (if you
|
||||
were watching Redis) would be zero -- the "cache" provides no speedup, only
|
||||
a leak.
|
||||
|
||||
## slow_consumer (`notifier`)
|
||||
|
||||
**Mechanism:** `services/notifier/consumer.py`. Sleeps 250ms per stream
|
||||
message before acking, inside the `notifier` consumer-group loop.
|
||||
|
||||
**Symptom:** the `notifier` consumer-group lag on `stream:transactions` grows
|
||||
under sustained load and drains slowly afterward, while the `audit` group
|
||||
(same stream, independent group) keeps up fine -- isolating the problem to
|
||||
one specific consumer, not the stream or producer.
|
||||
|
||||
## cold_cache (`insights`)
|
||||
|
||||
**Mechanism:** `services/insights/app.py`. The Redis summary cache TTL drops
|
||||
from 900s to 60s.
|
||||
|
||||
**Symptom:** repeated identical requests for the same account show periodic
|
||||
latency spikes roughly every 60s (a full recompute) instead of one initial
|
||||
spike followed by uniformly fast cache hits.
|
||||
|
||||
## poison_data (`ledger`)
|
||||
|
||||
**Mechanism:** `services/ledger/app.py` / `repository.py::find_null_currency_row`.
|
||||
Every `POST /internal/transactions` checks whether the account has any
|
||||
existing transaction with a `NULL` currency (one of the seeded malformed row
|
||||
varieties, §8). When the flag is on, that row's `currency` is accessed with no
|
||||
null guard (`.upper()` on `None`), which raises and surfaces as an unhandled
|
||||
HTTP 500. When off, the same check is guarded and does nothing.
|
||||
|
||||
**Symptom:** a specific, small set of accounts reliably 500 on every
|
||||
transaction attempt while the flag is on, and behave completely normally
|
||||
while it's off -- input data that was always there, suddenly fatal.
|
||||
|
||||
## chatty_health (`gateway`)
|
||||
|
||||
**Mechanism:** `services/gateway/orchestrator.py::_run_chatty_health_probe`.
|
||||
Before doing any real work, `POST /api/transactions` calls `GET /healthz`
|
||||
sequentially on all five downstreams, tagged with an `X-Ledgerline-Probe`
|
||||
header so the extra hits are distinguishable from routine health polling.
|
||||
|
||||
**Symptom:** every transaction request produces five extra, sequential
|
||||
health-check round trips before the real work even starts, inflating latency
|
||||
by a fixed per-request tax that has nothing to do with load or data size.
|
||||
@@ -0,0 +1,373 @@
|
||||
# Phase 0 — Technical Spec: "Ledgerline" base project
|
||||
|
||||
> **How to use this file:** paste the whole thing into your coding agent as the task
|
||||
> spec. It is written to be executed without further clarification. Read
|
||||
> §2 (Non-goals) yourself before you start — the deliberate omissions are the
|
||||
> point of the exercise.
|
||||
|
||||
---
|
||||
|
||||
## 1. Mission
|
||||
|
||||
Build **Ledgerline**, a deliberately imperfect microservice system that simulates a
|
||||
small payments/finance backend. It exists for one reason: to be a realistic subject
|
||||
for observability work in later phases. The business logic is fake and does not
|
||||
need to be correct, defensible, or secure. What it *does* need to be is
|
||||
**structurally realistic** — real network hops, real database queries, real async
|
||||
message passing, real contention, real latency distributions, and real failure modes.
|
||||
|
||||
The system must be runnable with `make up && make seed && make smoke` on a laptop.
|
||||
|
||||
---
|
||||
|
||||
## 2. Non-goals — read carefully
|
||||
|
||||
These are **hard constraints**, not preferences. Violating them destroys the value
|
||||
of the project.
|
||||
|
||||
- **DO NOT add any observability tooling.** No OpenTelemetry, no Prometheus client,
|
||||
no `/metrics` endpoint, no structured/JSON logging, no trace IDs, no spans, no
|
||||
APM agent, no Sentry. Later phases add all of this by hand; that is the lesson.
|
||||
- **DO NOT fix the deliberate pathologies in §7.** They are features. Do not add
|
||||
indexes, connection-pool tuning, batching, backoff, or caching beyond what is
|
||||
specified.
|
||||
- **DO NOT add authentication, authorization, rate limiting, or input hardening.**
|
||||
A single hardcoded demo user is fine.
|
||||
- **DO NOT write unit tests beyond a single smoke test.** Test coverage is not the
|
||||
deliverable.
|
||||
- **DO NOT use Kubernetes, Helm, or any orchestrator.** Docker Compose only.
|
||||
- **DO NOT abstract prematurely.** Duplication across services is acceptable and
|
||||
preferred over a clever shared framework.
|
||||
|
||||
The only logging permitted is Python's standard `logging` module with a plain-text
|
||||
formatter (§9).
|
||||
|
||||
---
|
||||
|
||||
## 3. Technology
|
||||
|
||||
| Concern | Choice |
|
||||
|---|---|
|
||||
| Language | Python 3.12 |
|
||||
| Web framework | FastAPI + `uvicorn` (one worker per service) |
|
||||
| HTTP client | `httpx` (async) |
|
||||
| DB | PostgreSQL 16, accessed via SQLAlchemy 2.0 async + `asyncpg` |
|
||||
| Cache / queue | Redis 7 (`redis-py` asyncio client) |
|
||||
| Async messaging | Redis **Streams** (not Celery, not RabbitMQ) |
|
||||
| Validation | Pydantic v2 |
|
||||
| Packaging | One shared `Dockerfile`, one image, `CMD` differs per service |
|
||||
| Orchestration | Docker Compose |
|
||||
| Schema | A single idempotent `db/schema.sql` — no Alembic |
|
||||
| Frontend | Plain HTML + CSS + vanilla JS, no build step, no framework |
|
||||
|
||||
Redis Streams is chosen deliberately: it has **no automatic context propagation**,
|
||||
so a later phase has to implement it manually. Do not swap it for something with
|
||||
built-in instrumentation.
|
||||
|
||||
---
|
||||
|
||||
## 4. Services
|
||||
|
||||
Seven services (six HTTP, one worker-only). All listen on `0.0.0.0`.
|
||||
|
||||
| # | Service | Port | Role | Stores |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `gateway` | 8000 | Public API + serves the frontend + orchestrates everything | Redis (flags) |
|
||||
| 2 | `ledger` | 8001 | Accounts, balances, transaction write path | Postgres |
|
||||
| 3 | `fraud` | 8002 | Risk scoring; calls `sanctions` | Redis (score cache) |
|
||||
| 4 | `sanctions` | 8003 | Fake third-party watchlist lookup — the "slow external vendor" | Postgres (read-only) |
|
||||
| 5 | `insights` | 8004 | Analytics over transaction history | Postgres (read-heavy) + Redis |
|
||||
| 6 | `notifier` | 8005 | HTTP API **and** a background Redis Stream consumer | Postgres |
|
||||
| 7 | `audit` | — | Worker only; second consumer group on the same stream | Postgres |
|
||||
|
||||
Every HTTP service exposes:
|
||||
|
||||
- `GET /healthz` → `{"status": "ok", "service": "<name>"}` — liveness, no dependency checks
|
||||
- `GET /readyz` → checks its own DB/Redis connection, returns 503 if unavailable
|
||||
- `GET /` → `{"service": "<name>", "version": "0.1.0"}`
|
||||
|
||||
---
|
||||
|
||||
## 5. Domain model (`db/schema.sql`)
|
||||
|
||||
```
|
||||
accounts (id UUID pk, holder_name text, currency char(3),
|
||||
balance_minor bigint, is_hot boolean default false,
|
||||
created_at timestamptz)
|
||||
|
||||
merchant_categories (id smallint pk, code text unique, label text,
|
||||
risk_weight numeric(3,2))
|
||||
|
||||
transactions (id UUID pk, account_id UUID, kind text, -- deposit|withdrawal|purchase
|
||||
amount_minor bigint, currency char(3),
|
||||
merchant_name text, category_id smallint,
|
||||
status text, -- pending|settled|rejected
|
||||
risk_score numeric(4,3), created_at timestamptz)
|
||||
|
||||
sanctions_entries (id serial pk, normalized_name text, country char(2),
|
||||
list_source text, added_at timestamptz)
|
||||
|
||||
notifications (id UUID pk, transaction_id UUID, channel text, -- email|push|sms
|
||||
status text, attempts int, sent_at timestamptz)
|
||||
|
||||
audit_events (id bigserial pk, transaction_id UUID, event_type text,
|
||||
payload jsonb, recorded_at timestamptz)
|
||||
```
|
||||
|
||||
**Index policy — important.** Create only these indexes:
|
||||
|
||||
- primary keys (implicit)
|
||||
- `accounts(holder_name)`
|
||||
- `sanctions_entries(normalized_name)`
|
||||
|
||||
Deliberately **omit** `transactions(account_id, created_at)` and
|
||||
`transactions(category_id)`. Put the `CREATE INDEX` statements for those in
|
||||
`db/optional_indexes.sql`, applied only by `make fix-index`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Request flows
|
||||
|
||||
### 6.1 `POST /api/transactions` — the primary flow
|
||||
|
||||
The trace this produces should be ~4 levels deep with one async fork.
|
||||
|
||||
```
|
||||
browser
|
||||
└─ gateway POST /api/transactions
|
||||
├─ ledger POST /internal/transactions (create as `pending`, row lock on account)
|
||||
├─ fraud POST /internal/score (risk scoring)
|
||||
│ └─ sanctions POST /internal/lookup (watchlist check — slow vendor)
|
||||
├─ ledger POST /internal/transactions/{id}/settle (or /reject if risk_score > 0.8)
|
||||
├─ redis XADD stream:transactions (fire-and-forget)
|
||||
│ ├─ notifier consumer group `notifier`
|
||||
│ └─ audit consumer group `audit`
|
||||
└─ insights GET /internal/accounts/{id}/summary (cached)
|
||||
```
|
||||
|
||||
Response body:
|
||||
|
||||
```json
|
||||
{
|
||||
"transaction_id": "…", "status": "settled", "risk_score": 0.12,
|
||||
"balance_minor": 481200, "summary": { "…": "…" },
|
||||
"request_id": "…", "elapsed_ms": 843
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 `GET /api/accounts/{id}/insights` — the analytics flow
|
||||
|
||||
`gateway → insights`. `insights` runs a 120-day aggregation over `transactions`.
|
||||
This is where the N+1 and the missing index bite.
|
||||
|
||||
### 6.3 `GET /api/accounts` — list demo accounts for the frontend dropdown
|
||||
|
||||
### 6.4 Admin
|
||||
|
||||
- `GET /api/admin/chaos` → current flag values
|
||||
- `POST /api/admin/chaos` → `{"flag": "slow_sanctions", "enabled": true}`
|
||||
- `POST /api/admin/chaos/profile` → `{"profile": "clean" | "light" | "nightmare" | "random"}`
|
||||
|
||||
Flags live in a Redis hash `chaos:flags` so every service reads the same state
|
||||
without a restart. Each service reads the hash on **every request** (a tiny bit of
|
||||
extra Redis load is fine and realistic).
|
||||
|
||||
---
|
||||
|
||||
## 7. The pathology catalogue
|
||||
|
||||
Each pathology lives in a clearly named `chaos.py` module inside its owning service,
|
||||
with a docstring naming it. All flags default **off**.
|
||||
|
||||
| Flag | Service | Behaviour when enabled |
|
||||
|---|---|---|
|
||||
| `slow_sanctions` | `sanctions` | Sleep for a log-normal delay: median ≈ 40 ms, p99 ≈ 1.8 s. Additionally, 1.5% of calls sleep 5 s (the caller's timeout is 3 s, so these surface as timeouts). |
|
||||
| `sanctions_errors` | `sanctions` | Return HTTP 503 for 3% of requests, chosen at random. |
|
||||
| `n_plus_one` | `insights` | Instead of a single `JOIN` on `merchant_categories`, loop over result rows and issue one `SELECT` per row. |
|
||||
| `pool_starvation` | `ledger` | Build the SQLAlchemy engine with `pool_size=2, max_overflow=0, pool_timeout=10`. (Default otherwise: `pool_size=20`.) |
|
||||
| `row_lock_contention` | `ledger` | Wrap the balance update in `SELECT … FOR UPDATE` and hold the transaction open for an extra 150 ms before committing. Only applies to accounts with `is_hot = true`. |
|
||||
| `blocking_event_loop` | `fraud` | Run the scoring routine as a synchronous CPU-bound loop (~120 ms of real work, e.g. repeated hashing) directly inside the `async def` handler, with no `run_in_executor` / `to_thread`. |
|
||||
| `retry_storm` | `gateway` | On any non-2xx from `fraud`, retry 3 times immediately — no backoff, no jitter, no circuit breaker. |
|
||||
| `no_batching` | `gateway` | Split the transaction into 1–5 synthetic "line items" and make one separate `fraud` call per item, sequentially, instead of one call. |
|
||||
| `unbounded_cache` | `insights` | Memoize summaries in a module-level `dict` keyed by `(account_id, timestamp_ms)` — effectively never hits, never evicts, grows forever. |
|
||||
| `slow_consumer` | `notifier` | Sleep 250 ms per stream message before acking, so the consumer group lag grows under load. |
|
||||
| `cold_cache` | `insights` | Set the Redis summary cache TTL to 60 s (instead of 900 s), producing periodic latency spikes. |
|
||||
| `poison_data` | `ledger` | Skip input validation, so the malformed seed rows (§8) cause unhandled exceptions → HTTP 500 on roughly 2% of requests touching them. |
|
||||
| `chatty_health` | `gateway` | On every `/api/transactions` call, first `GET /healthz` on all five downstreams sequentially before doing any real work. |
|
||||
|
||||
**Profiles**
|
||||
|
||||
- `clean` — all off.
|
||||
- `light` — `slow_sanctions`, `cold_cache`.
|
||||
- `nightmare` — all thirteen on.
|
||||
- `random` — pick a random subset of size 3–6, using a seed derived from
|
||||
`time.time()` and **not logged anywhere**. Store the active set in Redis under
|
||||
`chaos:flags` only, never write it to a file or print it. (This makes the later
|
||||
bottleneck-hunting exercise a genuine hunt.)
|
||||
|
||||
Write `docs/PATHOLOGIES.md` documenting each pathology, its mechanism, and the
|
||||
symptom a reader should expect. Do **not** reproduce or summarise its contents in
|
||||
your final chat response.
|
||||
|
||||
---
|
||||
|
||||
## 8. Seed data (`scripts/seed.py`)
|
||||
|
||||
Deterministic given `SEED=42` (settable via env). Idempotent: truncate then insert.
|
||||
|
||||
- **800 accounts.** Realistic-looking names. Currencies weighted 70% EUR, 20% USD,
|
||||
10% GBP. Balances log-normal, median ≈ 250 000 minor units.
|
||||
Exactly **5 accounts** flagged `is_hot = true` — these are the contention targets.
|
||||
- **45 merchant categories** with `risk_weight` between 0.01 and 0.95.
|
||||
- **250 000 transactions** spread over the last 120 days.
|
||||
- Timestamps follow a daily sinusoid (business-hours peak) plus weekend dip.
|
||||
- Amounts log-normal.
|
||||
- Skew: the 5 hot accounts own ~30% of all rows; the remaining 795 share the rest
|
||||
on a Zipf-ish distribution.
|
||||
- `status` 92% `settled`, 5% `rejected`, 3% `pending`.
|
||||
- **12 000 `sanctions_entries`.** ~40 of them must match names of seeded account
|
||||
holders, so a meaningful fraction of requests genuinely score as high risk.
|
||||
- **~2% malformed rows** distributed through `transactions`:
|
||||
- `currency` = `NULL`
|
||||
- negative `amount_minor` on a `deposit`
|
||||
- `merchant_name` containing 4-byte emoji and a NUL-ish escape sequence
|
||||
- `created_at` set 3 days in the future
|
||||
- `category_id` pointing at a non-existent category
|
||||
|
||||
Print a summary table at the end (row counts per table, hot account IDs, count of
|
||||
malformed rows). Seeding 250k rows must complete in under 60 seconds — use
|
||||
`COPY` / `executemany`, not per-row inserts.
|
||||
|
||||
---
|
||||
|
||||
## 9. Logging (the deliberately bad baseline)
|
||||
|
||||
Every service configures:
|
||||
|
||||
```python
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s %(message)s",
|
||||
)
|
||||
```
|
||||
|
||||
- Log lines are **plain text, human-prose style**: `logger.info(f"processing tx for account {account_id}, amount {amount}")`. Not JSON. Not key-value.
|
||||
- `gateway` generates an `X-Request-Id` (uuid4) per inbound request and forwards it
|
||||
as a header on every downstream `httpx` call. Each service logs it manually at the
|
||||
start of each handler. **This is the only correlation mechanism that exists.**
|
||||
It is intentionally weak — it does not survive the Redis Stream hop, it carries no
|
||||
timing, and it cannot express parent/child relationships.
|
||||
- Set `PYTHONUNBUFFERED=1` in the Dockerfile.
|
||||
- Log exceptions with `logger.exception(...)`, but do not add any error-tracking SDK.
|
||||
|
||||
---
|
||||
|
||||
## 10. Frontend
|
||||
|
||||
Served by `gateway` at `/` from `gateway/static/`. Three files: `index.html`,
|
||||
`app.js`, `style.css`. No bundler, no npm, no CDN dependencies.
|
||||
|
||||
Layout, top to bottom:
|
||||
|
||||
1. **Submit a transaction** — account dropdown (populated from `GET /api/accounts`),
|
||||
amount input, merchant name input, kind radio (deposit / withdrawal / purchase),
|
||||
Submit button.
|
||||
2. **Result panel** — pretty-printed JSON response, plus the browser-measured
|
||||
round-trip time in ms, plus the `request_id` shown in a monospace font that is
|
||||
easy to copy.
|
||||
3. **Load burst** — a button that fires *N* concurrent requests (N selectable:
|
||||
10 / 50 / 200) against random accounts, then renders a small table:
|
||||
count, success, errors, min / mean / p50 / p95 / max ms, measured client-side.
|
||||
This is a crude stand-in for a real load tool; a later phase replaces it with k6.
|
||||
4. **Chaos panel** — a checkbox per flag reflecting current state from
|
||||
`GET /api/admin/chaos`, toggling via `POST /api/admin/chaos`, plus four profile
|
||||
buttons (`clean`, `light`, `nightmare`, `random`).
|
||||
5. **Service health strip** — polls `/healthz` of all six HTTP services every 5 s
|
||||
via a `GET /api/health/all` aggregation endpoint on the gateway, showing a green
|
||||
or red dot per service.
|
||||
|
||||
Styling: dark, monospace-leaning, minimal. No frameworks. Roughly 200 lines of CSS.
|
||||
|
||||
---
|
||||
|
||||
## 11. Repository layout
|
||||
|
||||
```
|
||||
ledgerline/
|
||||
├─ Makefile
|
||||
├─ docker-compose.yml
|
||||
├─ Dockerfile # single image for all Python services
|
||||
├─ pyproject.toml # or requirements.txt — one dependency set
|
||||
├─ .env.example
|
||||
├─ README.md
|
||||
├─ db/
|
||||
│ ├─ schema.sql
|
||||
│ └─ optional_indexes.sql
|
||||
├─ scripts/
|
||||
│ ├─ seed.py
|
||||
│ └─ smoke.py
|
||||
├─ common/
|
||||
│ ├─ __init__.py
|
||||
│ ├─ settings.py # pydantic-settings, env-driven
|
||||
│ ├─ db.py # async engine/session factory
|
||||
│ ├─ redis_client.py
|
||||
│ ├─ http.py # httpx.AsyncClient factory, 3s timeout, X-Request-Id passthrough
|
||||
│ └─ logging_config.py # the plain-text logging setup from §9
|
||||
├─ services/
|
||||
│ ├─ gateway/ (app.py, chaos.py, orchestrator.py, static/)
|
||||
│ ├─ ledger/ (app.py, chaos.py, repository.py)
|
||||
│ ├─ fraud/ (app.py, chaos.py, scoring.py)
|
||||
│ ├─ sanctions/ (app.py, chaos.py, repository.py)
|
||||
│ ├─ insights/ (app.py, chaos.py, analytics.py)
|
||||
│ ├─ notifier/ (app.py, chaos.py, consumer.py)
|
||||
│ └─ audit/ (worker.py, consumer.py)
|
||||
└─ docs/
|
||||
└─ PATHOLOGIES.md
|
||||
```
|
||||
|
||||
`docker-compose.yml` defines: `postgres`, `redis`, and the seven services. Use
|
||||
`depends_on` with `condition: service_healthy` for postgres and redis. Mount the
|
||||
source read-write and run uvicorn with `--reload` so iteration is fast.
|
||||
|
||||
Reserve — but leave empty and unreferenced — a top-level `observability/` directory
|
||||
with a `.gitkeep`. Later phases fill it.
|
||||
|
||||
---
|
||||
|
||||
## 12. Makefile targets
|
||||
|
||||
| Target | Effect |
|
||||
|---|---|
|
||||
| `make up` / `make down` | compose up -d / down |
|
||||
| `make logs` | `docker compose logs -f` |
|
||||
| `make seed` | apply `db/schema.sql`, run `scripts/seed.py` |
|
||||
| `make reset` | `down -v`, `up`, `seed` |
|
||||
| `make smoke` | run `scripts/smoke.py` — one transaction, one insights call, assert 200s |
|
||||
| `make chaos-clean` / `chaos-light` / `chaos-nightmare` / `chaos-random` | set the profile |
|
||||
| `make fix-index` | apply `db/optional_indexes.sql` |
|
||||
| `make drop-index` | drop them again |
|
||||
|
||||
---
|
||||
|
||||
## 13. Acceptance criteria
|
||||
|
||||
The task is done when all of these hold on a clean checkout:
|
||||
|
||||
1. `make reset` completes in under 3 minutes and prints the seed summary table.
|
||||
2. `make smoke` passes with `chaos-clean`.
|
||||
3. The frontend loads at `http://localhost:8000`, the account dropdown is populated,
|
||||
and a submitted transaction returns a settled or rejected result.
|
||||
4. With `chaos-clean`, a 200-request burst from the frontend shows p95 under
|
||||
~250 ms and zero errors.
|
||||
5. With `chaos-nightmare`, the same burst shows p95 at least 5× worse and an error
|
||||
rate between 2% and 10% — degraded but not fully broken.
|
||||
6. Toggling any single flag via the chaos panel changes behaviour **without a
|
||||
restart**.
|
||||
7. `docker compose logs gateway` shows plain-text lines carrying `X-Request-Id`.
|
||||
8. `grep -ri "opentelemetry\|prometheus_client\|/metrics\|structlog" .` returns
|
||||
nothing.
|
||||
9. `docs/PATHOLOGIES.md` exists and documents all thirteen pathologies.
|
||||
|
||||
In your final message, report only: services started, row counts seeded, the
|
||||
p95/error-rate numbers you measured for `clean` vs `nightmare`, and anything you
|
||||
deviated from in this spec. Do not summarise `docs/PATHOLOGIES.md`.
|
||||
@@ -0,0 +1,508 @@
|
||||
# Phase 0a — Technical Spec: Ledgerline integration test suite
|
||||
|
||||
> **How to use this file:** paste it into the **test-authoring agent** together
|
||||
> with `phase-0-base-project-spec.md`. That agent writes only `tests/`.
|
||||
> A second agent then implements the services against the resulting suite.
|
||||
> See §1 for the workflow contract between them.
|
||||
|
||||
---
|
||||
|
||||
## 1. The two-agent workflow
|
||||
|
||||
### Agent A — test author (this spec)
|
||||
|
||||
- Reads `phase-0-base-project-spec.md` (the system contract) and this file.
|
||||
- Creates **only** these paths:
|
||||
```
|
||||
tests/** pytest.ini (or pyproject [tool.pytest])
|
||||
requirements-test.txt
|
||||
tests/README.md
|
||||
tests/RED-BASELINE.md
|
||||
```
|
||||
Plus the `test-*` targets appended to the `Makefile`.
|
||||
- Creates **nothing** under `services/`, `common/`, `db/`, `scripts/`,
|
||||
`observability/` or `loadtest/`. Not even a stub. If a test needs a fixture
|
||||
from the system, it asserts against the contract in §4 rather than importing
|
||||
application code.
|
||||
- **Imports nothing from the application.** Tests talk to the system over HTTP,
|
||||
Postgres, Redis, and the Docker CLI. No `from services.gateway import …`.
|
||||
This is what keeps them behavioural rather than implementation-coupled.
|
||||
- Runs the suite against an empty repo and confirms it is **red**, then writes
|
||||
`tests/RED-BASELINE.md` listing, per test, *why* it currently fails
|
||||
(connection refused / 404 / assertion / missing table). A test that passes
|
||||
against nothing is a broken test — find it and fix it.
|
||||
|
||||
### Agent B — implementer
|
||||
|
||||
- Reads `phase-0-base-project-spec.md` and treats `tests/` as a **read-only
|
||||
contract**.
|
||||
- **Must not modify, delete, skip, xfail, or loosen any test**, or change
|
||||
`pytest.ini`, markers, thresholds, or fixtures. If a test appears wrong,
|
||||
impossible, or contradictory: **stop and report it**, don't edit it.
|
||||
- Must implement each pathology by the mechanism described in
|
||||
`phase-0-base-project-spec.md` §7 — not as a generic `sleep()` tuned to make a
|
||||
timing assertion pass. Several assertions in §6 are deliberately built so that
|
||||
faking is harder than implementing (query counts, `EXPLAIN` plan shapes, Redis
|
||||
consumer lag, process RSS). Treat that as the intent everywhere, including
|
||||
where the assertion happens to be simpler.
|
||||
- Done when `make test` is green three consecutive runs.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design principles for this suite
|
||||
|
||||
These are the rules the tests must obey. They matter more than any individual
|
||||
test.
|
||||
|
||||
1. **Assert goals, not implementation.** No test may reference a function name,
|
||||
module path, class, SQL string, log format, or internal attribute. A test may
|
||||
reference the *public contract*: HTTP routes, status codes, JSON field names,
|
||||
table names, Redis stream and consumer-group names, chaos flag names.
|
||||
2. **Prefer assertions that are hard to satisfy without the real mechanism.**
|
||||
Wall-clock latency alone is fakeable with `sleep()`. Query counts from
|
||||
`pg_stat_statements`, `EXPLAIN` plan node types, Redis consumer-group lag,
|
||||
container RSS, and log-line counts on a *different service* are not. Where
|
||||
both are available, assert on both.
|
||||
3. **Measure differentially, never absolutely.** With one exception (§5.7),
|
||||
every pathology test compares flag-on against flag-off in the same run, on
|
||||
the same machine, and asserts a *ratio*. Absolute millisecond thresholds do
|
||||
not survive a different laptop.
|
||||
4. **Design for a noisy laptop.** Generous margins, warm-up requests discarded,
|
||||
median-of-N measurement runs. A test that fails when Slack is open is worse
|
||||
than no test.
|
||||
5. **Fast tests and slow tests are separate.** `make test-contract` must run in
|
||||
under 60 seconds. Pathology tests may take minutes and live behind their own
|
||||
marker.
|
||||
6. **Every test restores state.** A test that enables a chaos flag must reset
|
||||
the profile in teardown, even on failure.
|
||||
|
||||
---
|
||||
|
||||
## 3. Stack, layout, and execution
|
||||
|
||||
Tests run **on the host**, against the Compose stack on `localhost`. They do not
|
||||
run inside a container and do not use testcontainers — the stack is started by
|
||||
`make up` beforehand.
|
||||
|
||||
```
|
||||
tests/
|
||||
├─ README.md
|
||||
├─ RED-BASELINE.md
|
||||
├─ conftest.py # fixtures: clients, chaos control, seed manifest
|
||||
├─ harness/
|
||||
│ ├─ load.py # concurrent request driver + LatencyReport
|
||||
│ ├─ pg.py # pg_stat_statements deltas, EXPLAIN helpers
|
||||
│ ├─ streams.py # Redis XINFO GROUPS helpers
|
||||
│ └─ containers.py # docker stats / docker logs helpers
|
||||
├─ test_contract_health.py
|
||||
├─ test_contract_transactions.py
|
||||
├─ test_contract_insights.py
|
||||
├─ test_contract_async.py
|
||||
├─ test_contract_chaos.py
|
||||
├─ test_contract_frontend.py
|
||||
├─ test_seed_data.py
|
||||
├─ test_baseline_health.py
|
||||
├─ test_pathologies.py
|
||||
├─ test_nightmare.py
|
||||
└─ test_no_observability.py
|
||||
```
|
||||
|
||||
Dependencies (`requirements-test.txt`): `pytest`, `pytest-asyncio`,
|
||||
`pytest-timeout`, `httpx`, `psycopg[binary]`, `redis`. Docker access via
|
||||
`subprocess` calling the `docker` CLI — no SDK.
|
||||
|
||||
### Markers
|
||||
|
||||
| Marker | Runtime | Contents |
|
||||
|---|---|---|
|
||||
| `contract` | < 60 s | Functional behaviour, chaos-clean |
|
||||
| `seed` | < 20 s | Seed data integrity |
|
||||
| `baseline` | < 90 s | The clean system is genuinely healthy |
|
||||
| `pathology` | 5–12 min | One test per pathology, differential |
|
||||
| `slow` | — | Additional marker on the soak-ish ones (`unbounded_cache`, `cold_cache`) |
|
||||
| `guard` | < 5 s | The "no observability yet" assertions |
|
||||
|
||||
Configure `pytest.ini` with `--strict-markers`, `asyncio_mode = auto`, and a
|
||||
global `timeout = 300`.
|
||||
|
||||
### Make targets to append
|
||||
|
||||
| Target | Command |
|
||||
|---|---|
|
||||
| `test` | everything except `slow` |
|
||||
| `test-contract` | `-m "contract or seed or guard"` |
|
||||
| `test-baseline` | `-m baseline` |
|
||||
| `test-pathology` | `-m "pathology and not slow"` |
|
||||
| `test-slow` | `-m slow` |
|
||||
| `test-all` | no marker filter |
|
||||
|
||||
---
|
||||
|
||||
## 4. Contract additions required for testability
|
||||
|
||||
These extend `phase-0-base-project-spec.md`. Agent B must implement them; Agent A
|
||||
may assume them.
|
||||
|
||||
### 4.1 Seed manifest
|
||||
|
||||
`scripts/seed.py` writes `db/seed_manifest.json` **and** a `seed_manifest` table
|
||||
(single row, `data jsonb`) with the same content. Tests read the table, so they
|
||||
don't depend on a host file path.
|
||||
|
||||
```json
|
||||
{
|
||||
"seed": 42,
|
||||
"generated_at": "…",
|
||||
"counts": { "accounts": 800, "transactions": 250000,
|
||||
"merchant_categories": 45, "sanctions_entries": 12000,
|
||||
"malformed_transactions": 5013 },
|
||||
"hot_account_ids": ["…", "…", "…", "…", "…"],
|
||||
"cold_account_ids_sample": ["…"], // 50 non-hot accounts, for spread tests
|
||||
"sanctioned_account_ids": ["…"], // accounts whose holder matches the watchlist
|
||||
"poison_account_ids": ["…"], // accounts owning malformed rows
|
||||
"clean_account_id": "…" // no malformed rows, not hot, not sanctioned
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Response envelope
|
||||
|
||||
`POST /api/transactions` → **201** on success:
|
||||
|
||||
```json
|
||||
{ "transaction_id": "<uuid>", "status": "settled|rejected",
|
||||
"risk_score": 0.0-1.0, "balance_minor": 0,
|
||||
"summary": { }, "request_id": "<uuid>", "elapsed_ms": 0 }
|
||||
```
|
||||
|
||||
Errors return a consistent envelope with the appropriate status:
|
||||
|
||||
```json
|
||||
{ "error": "<machine_readable_code>", "detail": "<human string>",
|
||||
"request_id": "<uuid>" }
|
||||
```
|
||||
|
||||
Status mapping: `400` invalid input, `404` unknown account, `422` validation,
|
||||
`502` downstream returned an error, `504` downstream timed out, `500`
|
||||
unhandled — which is exactly what `poison_data` produces, and what §6.12 asserts.
|
||||
|
||||
### 4.3 Chaos API
|
||||
|
||||
- `GET /api/admin/chaos` → `{"profile": "<name|custom>", "flags": {"<flag>": bool, …}}`
|
||||
containing **all thirteen** flags, always.
|
||||
- `POST /api/admin/chaos` → `{"flag": "<name>", "enabled": bool}`; returns the
|
||||
full new state. Unknown flag → `400`.
|
||||
- `POST /api/admin/chaos/profile` → `{"profile": "clean|light|nightmare|random"}`;
|
||||
returns the full new state. For `random`, the response reports the flags —
|
||||
**but `make chaos-random` must not print them**, and the seed is never logged.
|
||||
- `GET /api/health/all` → `{"services": {"<name>": "ok"|"unhealthy", …}}` for all
|
||||
six HTTP services.
|
||||
|
||||
### 4.4 `pg_stat_statements`
|
||||
|
||||
The Postgres image must load `pg_stat_statements` (`shared_preload_libraries`)
|
||||
and the extension must be created by `db/schema.sql`. Tests use
|
||||
`pg_stat_statements_reset()` around a single request to count queries.
|
||||
|
||||
### 4.5 Deterministic seeding
|
||||
|
||||
`SEED` env var, default `42`. Same seed ⇒ identical manifest, identical row
|
||||
counts, identical hot/poison account IDs.
|
||||
|
||||
---
|
||||
|
||||
## 5. Goal-to-test traceability
|
||||
|
||||
Every test must map to one of these goals. Put the goal ID in the test's
|
||||
docstring. This is what keeps the suite about intent rather than mechanics.
|
||||
|
||||
| ID | Goal | Where |
|
||||
|---|---|---|
|
||||
| **G1** | A transaction traverses all seven services and returns a coherent result | §5.1 |
|
||||
| **G2** | The system is genuinely multi-service, not a collapsed monolith | §5.2 |
|
||||
| **G3** | Async work happens after the response, via two independent consumer groups | §5.3 |
|
||||
| **G4** | The seed data is large, skewed, and contains the intended poison rows | §5.4 |
|
||||
| **G5** | Chaos flags toggle at runtime without a restart | §5.5 |
|
||||
| **G6** | Each pathology measurably degrades the system when enabled | §6 |
|
||||
| **G7** | The clean baseline is actually healthy — otherwise G6 means nothing | §5.6 |
|
||||
| **G8** | Nightmare mode degrades but does not collapse | §5.7 |
|
||||
| **G9** | Failure is contained: timeouts fire, the gateway doesn't wedge | §5.8 |
|
||||
| **G10** | No observability tooling is present yet | §5.9 |
|
||||
| **G11** | The frontend and its supporting endpoints are served | §5.10 |
|
||||
|
||||
### 5.1 G1 — end-to-end transaction (`contract`)
|
||||
|
||||
- Submitting a valid transaction for `clean_account_id` returns 201 with the full
|
||||
envelope; `transaction_id` is a UUID; `status` ∈ {settled, rejected};
|
||||
`risk_score` ∈ [0,1]; `elapsed_ms` > 0.
|
||||
- The transaction is durable: a row with that ID exists in `transactions` with a
|
||||
terminal status within 2 s.
|
||||
- The balance moves in the correct direction for a deposit and for a withdrawal,
|
||||
by exactly `amount_minor`, when the transaction settles.
|
||||
- A transaction for a `sanctioned_account_ids` holder yields a materially higher
|
||||
`risk_score` than one for `clean_account_id` (assert the ordering, not a value).
|
||||
- `GET /api/accounts/{id}/insights` returns a summary for a seeded account.
|
||||
- Unknown account → 404 with the error envelope. Negative amount on a deposit →
|
||||
400 or 422. Missing required field → 422.
|
||||
|
||||
### 5.2 G2 — really seven services (`contract`)
|
||||
|
||||
- All six HTTP services answer `/healthz` on their own port with their own name.
|
||||
- `/readyz` returns 200 for all six.
|
||||
- `GET /api/health/all` reports all six as ok.
|
||||
- Stopping `sanctions` (`docker compose stop sanctions`) causes
|
||||
`GET /api/health/all` to report it unhealthy within 15 s, and
|
||||
`POST /api/transactions` to fail with 502 or 504 — **not** 200. Restart it in
|
||||
teardown and wait for ready.
|
||||
*(This is the test that catches a collapsed monolith: if transactions still
|
||||
succeed with `sanctions` stopped, the service isn't really in the path.)*
|
||||
|
||||
### 5.3 G3 — asynchronous work (`contract`)
|
||||
|
||||
- After a successful transaction, a row appears in `notifications` **and** in
|
||||
`audit_events` for that `transaction_id` within 10 s.
|
||||
- Both appear — proving two independent consumer groups, not one consumer doing
|
||||
both jobs.
|
||||
- `XINFO GROUPS stream:transactions` reports exactly two groups, named
|
||||
`notifier` and `audit`.
|
||||
- The API response returns **before** the notification row exists at least once
|
||||
across 20 attempts, proving the work is genuinely off the request path.
|
||||
*(Sample the DB immediately on response; assert at least one miss. Tolerant of
|
||||
a fast machine winning the race sometimes — hence 20 attempts, not 1.)*
|
||||
|
||||
### 5.4 G4 — seed data (`seed`)
|
||||
|
||||
Against the manifest and the database:
|
||||
|
||||
- Row counts match the manifest and the spec's targets within ±2%.
|
||||
- Exactly 5 accounts have `is_hot = true`, and they match `hot_account_ids`.
|
||||
- The 5 hot accounts own between 20% and 40% of all transactions.
|
||||
- Transactions span ≥ 110 distinct days.
|
||||
- Currency distribution is roughly 70/20/10 EUR/USD/GBP (±10 points each).
|
||||
- Status distribution ≈ 92/5/3 settled/rejected/pending (±5 points).
|
||||
- Malformed rows exist in all five described varieties: null currency, negative
|
||||
amount on a deposit, non-ASCII merchant name, future `created_at`, orphan
|
||||
`category_id`. Assert **at least one of each**, and 1%–4% overall.
|
||||
- ≥ 30 sanctions entries match seeded holder names.
|
||||
- Re-running the seed with the same `SEED` yields an identical manifest
|
||||
(compare the `counts` and `hot_account_ids`).
|
||||
- Neither `transactions(account_id, created_at)` nor `transactions(category_id)`
|
||||
exists in `pg_indexes` after a fresh seed.
|
||||
|
||||
### 5.5 G5 — chaos control (`contract`)
|
||||
|
||||
- `GET /api/admin/chaos` lists all thirteen flags by name.
|
||||
- Toggling a single flag changes only that flag.
|
||||
- Each profile sets the expected shape: `clean` → all false; `nightmare` → all
|
||||
true; `light` → exactly the two documented flags.
|
||||
- **No restart:** with `chaos-clean` set, record the `insights` latency; enable
|
||||
`n_plus_one` via the API; without restarting any container, the query count for
|
||||
the same request rises. Assert on the query-count delta (§6.3's helper), so
|
||||
this test proves live reconfiguration rather than timing noise.
|
||||
- Unknown flag name → 400.
|
||||
- Teardown restores `clean`.
|
||||
|
||||
### 5.6 G7 — the clean baseline is healthy (`baseline`)
|
||||
|
||||
With `chaos-clean`, after 20 discarded warm-up requests, drive 200 requests at
|
||||
concurrency 10 against random cold accounts:
|
||||
|
||||
- error rate == 0
|
||||
- p95 < 400 ms *(the one absolute threshold in the functional set; if it fails on
|
||||
a slow machine, that machine cannot run the pathology suite meaningfully
|
||||
either, and the failure message must say so)*
|
||||
- p99 / p50 < 5 — the clean system has no fat tail
|
||||
- `insights` for a cold account issues fewer than 10 SQL statements
|
||||
- Redis consumer lag returns to 0 within 15 s of the burst finishing
|
||||
|
||||
### 5.7 G8 — nightmare degrades but survives (`pathology`)
|
||||
|
||||
Same driver, `chaos-clean` then `chaos-nightmare`, 300 requests at concurrency 20
|
||||
each:
|
||||
|
||||
- nightmare p95 ≥ **5×** clean p95
|
||||
- nightmare error rate between **2% and 15%** — degraded, not dead
|
||||
- at least 80% of requests still return a valid envelope (either success or a
|
||||
structured error, never a connection reset or an empty body)
|
||||
- `GET /api/health/all` still responds within 5 s during the burst
|
||||
|
||||
### 5.8 G9 — failure containment (`contract`)
|
||||
|
||||
- With `sanctions_errors` at its maximum and `retry_storm` off, no request takes
|
||||
longer than 10 s. The 3 s downstream timeout must actually fire.
|
||||
- With `sanctions` stopped entirely, `POST /api/transactions` returns 502/504
|
||||
within 10 s, and the gateway still answers `/healthz` in under 1 s throughout.
|
||||
- No request ever returns a 200 with a `null` `transaction_id`.
|
||||
|
||||
### 5.9 G10 — no observability yet (`guard`)
|
||||
|
||||
Static and runtime assertions that the Phase 0 "before" state is intact:
|
||||
|
||||
- No file under `services/` or `common/` imports `opentelemetry`,
|
||||
`prometheus_client`, `structlog`, `sentry_sdk`, or `elasticapm`.
|
||||
- Neither `requirements*.txt` nor `pyproject.toml` lists them (excluding
|
||||
`requirements-test.txt`).
|
||||
- No service exposes `/metrics` (assert 404 on all six).
|
||||
- `docker compose logs gateway` emits plain text, not JSON: sample 20 lines and
|
||||
assert none parse as a JSON object.
|
||||
|
||||
> This test class is **expected to be deleted at the start of Phase 2**, not
|
||||
> maintained. Say so in its module docstring.
|
||||
|
||||
### 5.10 G11 — frontend (`contract`)
|
||||
|
||||
- `GET /` returns 200, `text/html`, and references `app.js` and `style.css`.
|
||||
- Those two assets return 200 with sensible content types.
|
||||
- `GET /api/accounts` returns ≥ 100 accounts, each with `id`, `holder_name`,
|
||||
`currency`, `balance_minor`.
|
||||
|
||||
No browser automation. Serving and shape only.
|
||||
|
||||
---
|
||||
|
||||
## 6. G6 — pathology tests
|
||||
|
||||
**One test per flag.** Each follows the same skeleton:
|
||||
|
||||
```
|
||||
1. set chaos-clean
|
||||
2. warm up (discard)
|
||||
3. measure → baseline
|
||||
4. enable exactly one flag via the API
|
||||
5. warm up (discard)
|
||||
6. measure → degraded
|
||||
7. assert the ratio, with the failure message printing both measurements
|
||||
8. teardown: chaos-clean
|
||||
```
|
||||
|
||||
Measurement runs the median of 3 for latency-based assertions. Failure messages
|
||||
must always include both raw numbers — a bare `assert ratio > 3` is useless at
|
||||
2 a.m.
|
||||
|
||||
| § | Flag | Primary signal (hard to fake) | Secondary signal | Assertion |
|
||||
|---|---|---|---|---|
|
||||
| 6.1 | `slow_sanctions` | — | p99/p50 of `POST /api/transactions` | degraded p99/p50 ≥ **6**, baseline p99/p50 ≤ 4; degraded p99 ≥ 5× baseline p99 |
|
||||
| 6.2 | `sanctions_errors` | count of 502/504 responses over 300 requests | — | degraded error rate ∈ [1%, 10%]; baseline == 0 |
|
||||
| 6.3 | `n_plus_one` | `pg_stat_statements` call delta across **one** insights request | latency ratio | degraded ≥ **50** statements; baseline ≤ 10; latency ratio ≥ 2 |
|
||||
| 6.4 | *missing index* (`make drop-index` / `fix-index`, not a chaos flag) | `EXPLAIN (ANALYZE, FORMAT JSON)` on the insights aggregation contains a `Seq Scan` on `transactions`; after `fix-index` it contains an `Index Scan`/`Bitmap Index Scan` | latency ratio | plan node types as stated; unindexed latency ≥ **3×** indexed. Teardown restores the dropped state. |
|
||||
| 6.5 | `pool_starvation` | throughput (req/s) at concurrency 5 vs 25 | p95 ratio | with the flag on, throughput at c=25 is ≤ **1.4×** throughput at c=5 (it plateaus); with it off, ≥ 2.5×. Also degraded p95 at c=25 ≥ 3× baseline p95 at c=25 |
|
||||
| 6.6 | `row_lock_contention` | 25 concurrent transactions against **one hot account** vs 25 spread across 25 cold accounts, both with the flag on | — | single-hot-account p95 ≥ **3×** spread p95; with the flag off the same comparison is ≤ 1.8× |
|
||||
| 6.7 | `blocking_event_loop` | wall-clock for 10 concurrent requests ÷ wall-clock for 1 request | — | degraded ratio ≥ **6** (serialised); baseline ratio ≤ 2.5 (concurrent) |
|
||||
| 6.8 | `retry_storm` | count of `sanctions` container log lines matching a lookup, per transaction, with `sanctions_errors` also on | — | degraded ≥ **2.0** lookups/transaction; baseline (retry off, errors on) ≤ 1.2. *Log-line counting is the only cross-service call counter available at Phase 0 — see the note below.* |
|
||||
| 6.9 | `no_batching` | `sanctions` lookup log lines per transaction, `sanctions_errors` **off** | latency ratio | degraded ≥ **2.0** lookups/transaction; baseline ≈ 1.0 (≤ 1.2); latency ratio ≥ 1.5 |
|
||||
| 6.10 | `unbounded_cache` (`slow`) | `insights` container RSS via `docker stats` before vs after 2000 requests with **distinct** query parameters, sampled 30 s after the burst ends | — | degraded RSS growth ≥ **20 MB** and still elevated at +30 s; baseline growth ≤ 8 MB |
|
||||
| 6.11 | `slow_consumer` | `XINFO GROUPS stream:transactions` → `lag`/`pending` for the `notifier` group after a 150-transaction burst | drain time | degraded lag ≥ **30** at burst end and still > 0 after 10 s; baseline drains to 0 within 10 s |
|
||||
| 6.12 | `cold_cache` (`slow`) | same insights request at 1 rps for 150 s; count samples > 3× the median | — | degraded ≥ **2** spikes (a 60 s TTL over 150 s); baseline ≤ 1 |
|
||||
| 6.13 | `poison_data` | request against each `poison_account_ids` entry | — | degraded produces ≥ 1 response with status **500**; baseline produces zero 500s across the same requests (4xx or 200 only) |
|
||||
| 6.14 | `chatty_health` | `/healthz` hits on downstream containers during a single transaction, counted from logs | — | degraded ≥ **5** healthz lines across downstreams per transaction; baseline == 0 |
|
||||
|
||||
**Note on log-line counting (6.8, 6.9, 6.14).** At Phase 0 the plain-text log is
|
||||
the only cross-service observability surface that exists — which is precisely the
|
||||
deficiency this whole project sets out to fix. Implement these helpers behind
|
||||
`harness/containers.py::count_log_lines(service, pattern, since)` so there is
|
||||
exactly one place to change. **From Phase 3 these assertions should be rewritten
|
||||
against structured logs, and from Phase 4 against span counts** — at which point
|
||||
they become both simpler and more robust. Note this in `tests/README.md`; it is a
|
||||
deliberate teaching artefact, not a shortcut.
|
||||
|
||||
**Anti-fake note for Agent B.** 6.3, 6.4, 6.10, 6.11 and 6.13 cannot be satisfied
|
||||
by inserting a `sleep()`: they require, respectively, real repeated queries, a
|
||||
real missing index, real unbounded allocation, real consumer slowness, and a real
|
||||
unhandled exception path. The timing-only assertions (6.1, 6.5–6.7, 6.9, 6.12)
|
||||
are stated as ratios for the same reason — passing them by faking is more work
|
||||
than implementing the mechanism described in the Phase 0 spec.
|
||||
|
||||
---
|
||||
|
||||
## 7. The measurement harness
|
||||
|
||||
`harness/load.py` provides one primitive:
|
||||
|
||||
```python
|
||||
async def drive(
|
||||
method: str, url_factory: Callable[[], tuple[str, dict]],
|
||||
total: int, concurrency: int,
|
||||
warmup: int = 20, timeout: float = 15.0,
|
||||
) -> LatencyReport
|
||||
```
|
||||
|
||||
`LatencyReport` carries `count, ok, errors, error_rate, status_counts,
|
||||
p50, p95, p99, max, throughput_rps, wall_seconds`, plus a `__str__` suitable for
|
||||
dropping straight into an assertion message.
|
||||
|
||||
Rules:
|
||||
|
||||
- Warm-up requests are issued and discarded before timing starts.
|
||||
- Percentiles from the raw sorted sample, nearest-rank. No interpolation, no
|
||||
averaging of percentiles across runs — **median of the p95s**, never the mean.
|
||||
- A closed-loop driver (fixed concurrency) is used deliberately for its
|
||||
simplicity here. Note in `tests/README.md` that this under-measures latency
|
||||
under saturation (coordinated omission) and that Phase 6 replaces it with k6's
|
||||
arrival-rate executors. Don't try to fix it in the test suite.
|
||||
- `measure_median(n=3, fn)` helper for the pathology tests.
|
||||
|
||||
`harness/pg.py`:
|
||||
|
||||
```python
|
||||
@contextmanager
|
||||
def statement_count(dsn) -> Counter # resets pg_stat_statements, yields, returns delta
|
||||
def explain(dsn, sql, params) -> dict # EXPLAIN (ANALYZE, FORMAT JSON)
|
||||
def plan_node_types(plan: dict) -> set[str]
|
||||
```
|
||||
|
||||
`harness/streams.py`: `group_lag(group)`, `wait_for_drain(group, timeout)`.
|
||||
|
||||
`harness/containers.py`: `rss_mb(service)`, `count_log_lines(service, pattern,
|
||||
since)`, `stop(service)`, `start(service)`, `wait_ready(service, timeout)`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Fixtures (`conftest.py`)
|
||||
|
||||
| Fixture | Scope | Purpose |
|
||||
|---|---|---|
|
||||
| `stack_up` | session | Assert the Compose stack is running and all six services are ready; **fail fast with a clear message** ("run `make up && make seed` first") rather than timing out |
|
||||
| `seeded` | session | Assert the seed manifest exists and counts are non-zero; fail fast otherwise |
|
||||
| `manifest` | session | Parsed seed manifest |
|
||||
| `api` | function | `httpx.AsyncClient` against the gateway, 15 s timeout |
|
||||
| `dsn` | session | Postgres DSN from env |
|
||||
| `redis_client` | session | — |
|
||||
| `chaos` | function | `chaos.set_profile(name)`, `chaos.set_flag(name, bool)`, `chaos.get()`. **Autouse teardown resets to `clean`**, even on failure |
|
||||
| `fresh_balances` | function | Snapshot and restore the balances of accounts a test touches, so balance assertions don't drift across runs |
|
||||
|
||||
Every environment value (ports, DSN, Redis URL) comes from env with the Compose
|
||||
defaults as fallbacks. No hardcoded values buried in test bodies.
|
||||
|
||||
---
|
||||
|
||||
## 9. `tests/README.md` — what Agent A must write
|
||||
|
||||
- How to run each marker set, and the prerequisite (`make up && make seed`).
|
||||
- The measurement methodology and why ratios are used rather than absolutes.
|
||||
- The known-flaky list: 6.10 and 6.12 are the least stable; document the retry
|
||||
strategy and what a genuine failure looks like versus noise.
|
||||
- The coordinated-omission caveat on the closed-loop driver.
|
||||
- The Phase 3/4 migration note for the log-counting helpers.
|
||||
- An explicit statement that `test_no_observability.py` is deleted at Phase 2.
|
||||
|
||||
---
|
||||
|
||||
## 10. Acceptance criteria
|
||||
|
||||
For **Agent A**:
|
||||
|
||||
1. `pytest --collect-only` succeeds and every test carries exactly one goal ID in
|
||||
its docstring and at least one marker.
|
||||
2. `grep -rn "from services\|from common\|import services\|import common" tests/`
|
||||
returns nothing.
|
||||
3. Every §5 and §6 item has a corresponding test; `tests/README.md` contains the
|
||||
goal-to-test table.
|
||||
4. The suite is **red** against an empty repo, and `tests/RED-BASELINE.md`
|
||||
records the failure reason per test. No test passes vacuously.
|
||||
5. `make test-contract` completes (failing) in under 60 s.
|
||||
|
||||
For **Agent B**:
|
||||
|
||||
1. `make test` green three consecutive runs.
|
||||
2. `make test-slow` green (may be run once).
|
||||
3. `git diff --stat tests/` against Agent A's commit is empty.
|
||||
4. `docs/PATHOLOGIES.md` written, and not summarised in the final response.
|
||||
@@ -0,0 +1,377 @@
|
||||
# Observability & APM — Delivery Roadmap for Babak's Task
|
||||
|
||||
*Part of `docs/roadmap/` — platform background and reading guidance shared by
|
||||
every phase. Per-phase detail lives in `phase-0.md` through `phase-10.md`;
|
||||
[`summary.md`](summary.md) gives the one-page overview and status of all of
|
||||
them.*
|
||||
|
||||
Ten phases. Each one: concepts first, then the build, then a checkable exit
|
||||
condition. Total time-boxed at roughly **4–6 weeks part-time** (evenings +
|
||||
weekends). Nothing here is theory for theory's sake — every concept block is
|
||||
followed immediately by the place it lands in the code.
|
||||
|
||||
---
|
||||
|
||||
## Read this first — the four questions blocking you
|
||||
|
||||
### "What *is* an APM? Do you install it, use it, implement it, or build it?"
|
||||
|
||||
APM (Application Performance Monitoring) is a **product category and a capability**,
|
||||
not a protocol or a single tool. A system has APM when you can answer, for
|
||||
production traffic: what's the request rate, error rate and latency distribution
|
||||
per endpoint; which services call which; where did *this specific* request spend its
|
||||
time; which database query or external call was slow; and what error was thrown,
|
||||
with the stack.
|
||||
|
||||
The verb depends on the vendor model:
|
||||
|
||||
- **Commercial APM (Datadog, New Relic, Dynatrace, AppDynamics, Elastic APM).** You
|
||||
*install* an agent that attaches to your runtime and ships to their backend. Mostly
|
||||
a deployment exercise. Turnkey, expensive, proprietary.
|
||||
- **Grafana stack — what Babak asked for.** There is no product called "Grafana APM"
|
||||
that you install. You **assemble** the capability from four layers:
|
||||
|
||||
| Layer | What you do | Component |
|
||||
|---|---|---|
|
||||
| Instrumentation | *Implement* — this is code you write | OpenTelemetry SDK inside your services |
|
||||
| Pipeline | *Deploy + configure* | OpenTelemetry Collector (or Grafana Alloy, Grafana's own OTel Collector distribution) |
|
||||
| Storage | *Deploy* | Tempo (traces), Loki (logs), Prometheus or Mimir (metrics) |
|
||||
| Query & UI | *Configure* | Grafana + provisioned datasources with cross-links |
|
||||
|
||||
So: you **implement** instrumentation, you **deploy** the pipeline and stores, you
|
||||
**configure** the correlation. The "APM" is the emergent property of all four
|
||||
working together.
|
||||
|
||||
The distinction that actually matters: APM data is generated *inside* the
|
||||
application, keyed to *individual requests*, and *correlated* across signals.
|
||||
Infrastructure monitoring (CPU, disk, pod restarts) is generated *outside* the
|
||||
application and aggregated. You need both; only the first is APM.
|
||||
|
||||
### "What's the difference between Logs, Metrics, Events and Traces?"
|
||||
|
||||
The clean way to hold this in your head is: **an event is the atom; the other three
|
||||
are shapes you give it.**
|
||||
|
||||
| | What it is | Cost shape | Answers |
|
||||
|---|---|---|---|
|
||||
| **Event** | A structured record of one thing that happened, at one instant, with arbitrary attributes attached | Grows with traffic × attribute count | "What exactly happened here?" |
|
||||
| **Log** | An event rendered for storage/search. A *structured* log is literally an event; an unstructured log is an event you threw the schema away from | Grows with traffic | "Why did it fail?" |
|
||||
| **Metric** | A pre-aggregated numeric measurement over many events, sliced by a small set of labels | Grows with *label cardinality*, ~flat in traffic | "Is it broken? How much? Since when?" |
|
||||
| **Trace** | A causally linked tree of events that each have a *duration* and a parent — a span tree spanning process boundaries | Grows with traffic × span count | "Where did the time go, across services?" |
|
||||
|
||||
Three consequences worth internalising now:
|
||||
|
||||
1. **Metrics are cheap in traffic but expensive in cardinality.** Doubling your
|
||||
request rate barely changes Prometheus's memory; adding `user_id` as a label
|
||||
can kill it. Logs and traces are the opposite: cheap in dimensions, expensive
|
||||
in volume. This asymmetry drives every design decision later.
|
||||
2. **Traces are events with two extra fields:** a duration and a parent pointer.
|
||||
That's the whole trick. Everything else (service maps, critical paths, span
|
||||
metrics) is derived from those two fields.
|
||||
3. In OpenTelemetry, "events" aren't a fourth top-level signal you export
|
||||
separately. They show up as **span events** (a timestamped annotation attached
|
||||
to a span — this is how `record_exception` works) and as **log records carrying
|
||||
an `event.name` attribute**; the Events API is built on the Logs data model.
|
||||
When Babak says M.E.L.T., he most likely means "logs plus the discrete
|
||||
business/domain events you care about" — check with him, but treat it as
|
||||
*structured logs and span events*, not a separate pipeline.
|
||||
|
||||
### "Do I learn OpenTelemetry first, or Loki/Prometheus/Tempo first?"
|
||||
|
||||
**OpenTelemetry first, without much doubt.**
|
||||
|
||||
Loki, Tempo, Prometheus and Mimir are *destinations*. They're databases with query
|
||||
languages. OpenTelemetry is the **producer-side data model and wire protocol** — it's
|
||||
the only part of this that touches your Python code, and it's the part that
|
||||
determines what the destinations can possibly show you. If you learn Loki first you
|
||||
learn Loki's label model and its promtail-isms, most of which don't transfer. If you
|
||||
learn OTel first, every backend becomes "point OTLP at a different URL."
|
||||
|
||||
One exception: learn **Prometheus's data model** (labels, series, cardinality) fairly
|
||||
early, in parallel with OTel metrics — because it *constrains what you're allowed to
|
||||
emit*. PromQL, LogQL and TraceQL are query languages; learn them when you need them
|
||||
(Phases 4–5), not up front.
|
||||
|
||||
Filebeat and Logstash are a **different world entirely** (Elastic's). They are not
|
||||
part of the Grafana stack and you do not need them for this task. Phase 9 covers
|
||||
them as a judgment question, which is exactly how Babak framed them.
|
||||
|
||||
### Verdict on the OneUptime article you found
|
||||
|
||||
**Yes, it's valuable — but not yet.** It's a *configuration cheat-sheet*, not a
|
||||
teaching resource. Read it in **Phase 4/5**, not now; it assumes you already
|
||||
understand spans, span metrics and PromQL.
|
||||
|
||||
What's genuinely good in it, and rare to find in one place:
|
||||
|
||||
- The `spanmetrics` and `servicegraph` **connectors** in the Collector — this is how
|
||||
you get RED metrics and a service map "for free" from traces you already emit.
|
||||
This is the single highest-leverage config in the whole article.
|
||||
- The Tempo datasource provisioning YAML with `tracesToLogsV2`, `tracesToMetrics`,
|
||||
`serviceMap` and `nodeGraph` — the click-through correlation that makes Grafana
|
||||
feel like an APM.
|
||||
- `exemplarTraceIdDestinations` on the Prometheus datasource — jump from a latency
|
||||
spike on a graph straight into the trace that caused it.
|
||||
- The tail-sampling policy block (keep all errors, keep all slow traces, sample the
|
||||
rest).
|
||||
|
||||
Two caveats:
|
||||
|
||||
- Its Collector config exports logs via the **`loki` exporter**. Loki now accepts
|
||||
**OTLP natively** on its own endpoint, and that is the current recommended path —
|
||||
fewer moving parts, better attribute mapping. Use the native OTLP route.
|
||||
- It's a vendor blog (OneUptime sells a competing product). The Grafana content is
|
||||
accurate, but treat the official OpenTelemetry and Grafana docs as the source of
|
||||
truth for anything spec-level.
|
||||
- **Its `spanmetrics` / `servicegraph` Collector connectors do not apply to your
|
||||
stack.** See the next section — your platform generates those metrics in Tempo
|
||||
instead. The article's PromQL is still useful, but the metric names differ.
|
||||
|
||||
---
|
||||
|
||||
## The platform Babak gave you (`observability-ansible`)
|
||||
|
||||
This changes *where* your telemetry goes and *how the pipeline is configured*. It
|
||||
does not change the learning arc, the app, or the test suite. Read this before
|
||||
Phase 1; it rewrites parts of Phases 1, 3, 4, 5, 7 and 9.
|
||||
|
||||
### What the repo is
|
||||
|
||||
An Ansible project that deploys **two independent observability backends** on one
|
||||
Ubuntu/Debian host via Docker Compose:
|
||||
|
||||
1. **ELK + Grafana** — Filebeat → Logstash → Elasticsearch → Kibana/Grafana
|
||||
2. **Grafana stack** — Prometheus + Alertmanager + Loki + Tempo + Grafana, with
|
||||
**Grafana Alloy** as the collector and node-exporter for host metrics
|
||||
|
||||
It contains **no application code and no dashboards**. It is the *platform half* of
|
||||
Babak's task, handed to you pre-built. Your half is everything that produces
|
||||
telemetry and everything that reads it.
|
||||
|
||||
Deployment: `ansible-playbook playbooks/grafana-stack.yml` and
|
||||
`playbooks/elk-grafana.yml`, idempotent, versions pinned in `group_vars/all.yml`
|
||||
(Grafana 12.1.1, Prometheus v3.5.0, Loki 3.5.3, Tempo 2.8.2, Alloy v1.10.2,
|
||||
Elastic 9.1.3). Everything lands under `/opt/observability/`.
|
||||
|
||||
### The architecture
|
||||
|
||||
Per `diagrams/02-grafana-lgtm-operational.svg`, the platform is five columns:
|
||||
**Workloads & Hosts → Collect & Export → Store & Query → Alert Pipeline →
|
||||
Explore & Correlate**, with three input classes feeding it.
|
||||
|
||||
```
|
||||
App metrics ──/metrics, Prometheus exposition──┐
|
||||
Container logs ──Docker socket, tail──▶ Alloy ──push API──▶ Loki
|
||||
Instrumented apps ──OTLP 4317/4318────────▶ Alloy ──OTLP/gRPC──▶ Tempo
|
||||
("spans + resource attrs") │
|
||||
Node Exporter :9100 ──────scrape──────┐ │
|
||||
▼ │
|
||||
Prometheus ◀──remote_write────────┘
|
||||
(TSDB, PromQL, rules) span metrics + exemplars
|
||||
│
|
||||
firing alerts ▼
|
||||
Alertmanager ──route──▶ Receivers
|
||||
(group, dedupe) (Slack, Email, PagerDuty,
|
||||
│ webhook, SMTP)
|
||||
Grafana ◀──query + response (PromQL / LogQL / TraceQL)──▶ all four
|
||||
Correlation key: trace_id / exemplars
|
||||
```
|
||||
|
||||
Note what the diagram says about the OTLP box: **"spans + resource attrs"**. Traces
|
||||
only. And app metrics are labelled **"/metrics endpoint, Prometheus exposition"** —
|
||||
not OTLP. The diagram and the code agree on this; it is a design decision, not an
|
||||
oversight. Plan your instrumentation around it.
|
||||
|
||||
### One place the diagram and the code disagree
|
||||
|
||||
The diagram draws app metrics as `App metrics → scrape → Alloy → /metrics →
|
||||
Prometheus`. The shipped config does not do that:
|
||||
|
||||
- `config.alloy.j2` contains **no `prometheus.scrape` component at all** — Alloy
|
||||
collects logs and receives traces, nothing else.
|
||||
- `prometheus.yml.j2` scrapes `prometheus`, `node-exporter`, `loki`, `tempo`,
|
||||
`alloy:12345`, `grafana`, then loops over `prometheus_extra_scrape_jobs`. So your
|
||||
app targets are scraped by **Prometheus directly**, bypassing Alloy entirely. The
|
||||
green `/metrics` arrow from Alloy to Prometheus in the diagram is Prometheus
|
||||
scraping Alloy's **own** self-monitoring endpoint, not your app's metrics
|
||||
passing through it.
|
||||
|
||||
Neither is wrong; the diagram shows the intended architecture and the code
|
||||
implements a simpler one. Closing that gap — adding `prometheus.scrape` +
|
||||
`prometheus.remote_write` to Alloy so metrics really do flow through the collector —
|
||||
is a genuinely good Phase 4 exercise, and **noticing the discrepancy at all is worth
|
||||
mentioning to Babak.** Spotting that a diagram and its implementation have drifted is
|
||||
exactly the skill this task is meant to build.
|
||||
|
||||
### Four things in it that will cost you a day if you don't know them
|
||||
|
||||
**1. Alloy's OTLP receiver only forwards traces.** The config is literally:
|
||||
|
||||
```
|
||||
otelcol.receiver.otlp "default" {
|
||||
grpc { endpoint = "0.0.0.0:4317" }
|
||||
http { endpoint = "0.0.0.0:4318" }
|
||||
output {
|
||||
traces = [otelcol.exporter.otlp.tempo.input]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
There is **no `metrics` output and no `logs` output**. OTLP metrics and OTLP logs
|
||||
your app sends are accepted and then **silently discarded** — no error, no warning,
|
||||
just nothing arriving in Grafana. This is the single most likely thing to eat an
|
||||
evening.
|
||||
|
||||
The diagram confirms this is deliberate: the OTLP input is labelled "spans +
|
||||
resource attrs", and metrics are drawn as a separate `/metrics` Prometheus-exposition
|
||||
input. So you have three options, and you should understand all three:
|
||||
|
||||
- *Follow the platform's grain (simplest):* expose a Prometheus `/metrics` endpoint
|
||||
per service and register them in `prometheus_extra_scrape_jobs` in
|
||||
`group_vars/all.yml`. Prometheus scrapes your services directly. Logs continue to
|
||||
arrive via stdout scraping.
|
||||
- *Implement what the diagram draws:* add `prometheus.scrape` +
|
||||
`prometheus.remote_write` components to Alloy so app metrics flow through the
|
||||
collector, as the architecture intends. Buys you central relabeling and one place
|
||||
to change collection policy.
|
||||
- *Go all-OTLP:* add `metrics` and `logs` outputs to the receiver, plus
|
||||
`otelcol.exporter.prometheus` → `prometheus.remote_write` and
|
||||
`otelcol.exporter.loki` → `loki.write`. One protocol for all three signals — the
|
||||
architecture you'd argue for on a greenfield platform, and the furthest from what
|
||||
ships.
|
||||
|
||||
Do the first to get moving, then do at least one of the others as a deliberate
|
||||
exercise and write up the trade-off. That comparison is worth more than any single
|
||||
answer.
|
||||
|
||||
**2. Span metrics come from Tempo, not from the collector.** `tempo.yml.j2` enables:
|
||||
|
||||
```
|
||||
metrics_generator:
|
||||
storage:
|
||||
remote_write:
|
||||
- url: http://prometheus:9090/api/v1/write
|
||||
send_exemplars: true
|
||||
overrides:
|
||||
defaults:
|
||||
metrics_generator:
|
||||
processors: [service-graphs, span-metrics, local-blocks]
|
||||
```
|
||||
|
||||
So RED metrics, the service graph, and exemplars already exist — generated inside
|
||||
Tempo and remote-written into Prometheus (which runs with
|
||||
`--web.enable-remote-write-receiver`). **Do not add `spanmetrics`/`servicegraph`
|
||||
connectors to Alloy** — you'd be generating the same series twice under different
|
||||
names. The metric names differ from the OneUptime article's; go find the actual
|
||||
names in Prometheus before writing any PromQL. (`local-blocks` also gives you
|
||||
TraceQL metrics queries in Tempo — a bonus worth exploring in Phase 8.)
|
||||
|
||||
**3. Log→trace correlation depends on an exact, case-sensitive regex.** The Loki
|
||||
datasource is provisioned with:
|
||||
|
||||
```
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
|
||||
name: TraceID
|
||||
```
|
||||
|
||||
Your log lines must contain **lowercase** `trace_id` (or `traceid` / `trace-id`).
|
||||
This matters more than it looks: OpenTelemetry's `opentelemetry-instrumentation-logging`
|
||||
auto-injection emits **`otelTraceID`**, which does **not** match that pattern.
|
||||
So on this platform, the structlog processor emitting a lowercase `trace_id` field
|
||||
isn't a stylistic preference — it's a requirement. Same for exemplars: the Prometheus
|
||||
datasource declares `exemplarTraceIdDestinations: name: trace_id`. The diagram states
|
||||
this outright in its legend: **"Correlation key: trace_id / exemplars."** Everything
|
||||
in the Explore & Correlate column hangs off that one field name.
|
||||
|
||||
**4. There is no dashboard provisioning at all.** The Grafana provisioning mount
|
||||
contains only `datasources/`. Every dashboard is yours to add — which means Babak's
|
||||
"create dashboards as code, destroy and recreate in seconds" requirement is entirely
|
||||
your deliverable, and the natural home for it is *inside this Ansible role*.
|
||||
|
||||
### What this changes, phase by phase
|
||||
|
||||
| Phase | Impact |
|
||||
|---|---|
|
||||
| 0 / 0a | **None.** App and test suite are unaffected. |
|
||||
| 1 | Replaced. Deploy the real stack with Ansible instead of running a single container. |
|
||||
| 2 | Endpoint changes to Alloy; tracing work itself is identical. |
|
||||
| 3 | Rewritten. OTLP logs go nowhere; stdout scraping is the path. Lowercase `trace_id` is mandatory. |
|
||||
| 4 | Rewritten. Tempo generates span metrics; app metrics need a scrape endpoint or an Alloy extension. |
|
||||
| 5 | **Upgraded.** Dashboards-as-code becomes an Ansible deliverable — a better answer than a Makefile target. |
|
||||
| 6 | None. |
|
||||
| 7 | Choice added: Prometheus rules + Alertmanager (the platform's grain) vs Grafana unified alerting. |
|
||||
| 8 | None, plus TraceQL metrics from `local-blocks`. |
|
||||
| 9 | **Massively upgraded.** You now have a real Filebeat → Logstash → Elasticsearch pipeline to run, not a thought experiment. |
|
||||
| 10 | Demo now includes `ansible-playbook` rebuilding the platform. |
|
||||
|
||||
### Where to run it
|
||||
|
||||
The README's own sizing is 2 vCPU / 4 GB for the Grafana stack and 4 vCPU / 8 GB for
|
||||
ELK. Your single-node k8s box is 4 CPU / 8 GB total — running ELK there alongside
|
||||
everything else will hurt. Use a separate VM (a cheap cloud instance or a local
|
||||
Multipass/UTM VM) as the Ansible target, and keep Ledgerline on your laptop pointing
|
||||
at it. That also makes the deployment *real* rather than a localhost shortcut, which
|
||||
is the point of it being Ansible.
|
||||
|
||||
If you do colocate, note that Alloy's `loki.source.docker` reads the host's Docker
|
||||
socket and will scrape **every** container's stdout, including Ledgerline's,
|
||||
automatically — convenient, and worth knowing before you wonder where those logs
|
||||
came from.
|
||||
|
||||
### One thing to confirm with Babak
|
||||
|
||||
Is this repo a **platform you must consume as-is**, or a **starting point you may
|
||||
modify**? It's in your own Gitea and nothing marks it read-only, so this roadmap
|
||||
assumes you own it and will edit `config.alloy.j2`, add a dashboards provider, and
|
||||
extend `group_vars`. If Babak intends it as a fixed platform, several
|
||||
recommendations below change — ask before Phase 3.
|
||||
|
||||
---
|
||||
|
||||
## Your resource spine
|
||||
|
||||
Pick **one paid course as a backbone** and use the free material as targeted
|
||||
supplements. Both Udemy options below ship captions, which matters for your
|
||||
subtitle-to-notes workflow.
|
||||
|
||||
**Backbone — choose one:**
|
||||
|
||||
| Resource | Format | Length | Cost | Best for |
|
||||
|---|---|---|---|---|
|
||||
| *Observability with Grafana, Prometheus, Loki, Alloy and Tempo* (Aref Karimi, Udemy) | Video | ~10 h, 132 lectures | Paid (often ~€15 on sale) | The **Grafana-stack backbone**. Long-running bestseller, updated Jan 2026, covers Prometheus, PromQL, Grafana dashboards, Loki, Tempo, Alloy, Mimir, and OTel. Best fit for Phases 3–5. Comes with browser-based Killercoda playgrounds. |
|
||||
| *OpenTelemetry for Observability: The Complete Course* (Udemy) | Video | ~7.5 h, 113 lectures | Paid | The **OTel backbone**. Instrumentation → context propagation → Prometheus/Loki/Tempo/Grafana → Kubernetes. Closest single course to Babak's exact task. Best fit for Phases 2–4. |
|
||||
|
||||
If budget allows, buy both — they overlap by maybe 30% and the second half of each
|
||||
is different. If only one: take the **OpenTelemetry** one, because instrumentation
|
||||
is the part you have to *write*, and Grafana's own docs are excellent for the rest.
|
||||
|
||||
**Free, and worth your time:**
|
||||
|
||||
| Resource | Format | Length | Use in |
|
||||
|---|---|---|---|
|
||||
| freeCodeCamp — *OpenTelemetry Course* (Ania Kubów) | YouTube | ~1 h | Phase 1. Concepts, M.E.L.T., context propagation. **Note: the code is Node.js**, not Python — watch it for the model, not the syntax. |
|
||||
| Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025) | YouTube talk | ~30 min | Phase 2. The best Python-specific overview, with a companion repo. |
|
||||
| Jessica Garson — *Introduction to OpenTelemetry with Django* (DjangoCon US 2024) | YouTube talk | ~30 min | Phase 2. Auto- vs manual-instrumentation side by side. |
|
||||
| **OpenTelemetry Demo ("Astronomy Shop")** — `github.com/open-telemetry/opentelemetry-demo`, and Grafana's fork `grafana/lgtm-otel-demo` | Running code | — | **Every phase.** This is your "how professionals do it" reference. ~25 containers, polyglot, real auto+manual instrumentation, a Collector config you can steal from, and a feature-flag service for injecting failures. Run it once early, then keep it open in a second terminal as a lookup. |
|
||||
| `grafana/docker-otel-lgtm` Docker image | Tool | — | Phase 1. Grafana + Prometheus + Tempo + Loki + Pyroscope + Collector in **one container** with OTel defaults pre-wired. Removes all backend setup friction so you can focus on instrumentation. |
|
||||
| Official OpenTelemetry docs — Concepts, Python instrumentation, Semantic Conventions, Sampling | Docs | — | Phases 1–4. The neutral source of truth. |
|
||||
| Grafana docs — *Observability as code* / Foundation SDK; k6 docs | Docs | — | Phases 5–6 |
|
||||
| **Grafana Alloy docs** — components reference, especially `otelcol.receiver.otlp`, `otelcol.exporter.*`, `loki.source.docker`, `prometheus.remote_write` | Docs | — | Phases 1–4. **Required** — your platform uses Alloy, not the vanilla Collector, and its config language is component-based, not YAML pipelines. |
|
||||
| **Grafana Tempo docs** — *Metrics-generator* and *TraceQL* | Docs | — | Phases 4, 8. This is where your RED metrics and service graph actually come from. |
|
||||
| Elastic docs — Filebeat inputs/modules, Logstash grok filter | Docs | — | Phase 9, now hands-on |
|
||||
| Google SRE Workbook — *Implementing SLOs*, *Alerting on SLOs*, *Error Budget Policy* | Free book chapters | ~2 h total | Phase 7 |
|
||||
| Google SRE — *The Art of SLOs* workshop (slides + participant & facilitator handbooks, all public PDFs) | Slides/PDF + exercises | ~2 h | Phase 7. Self-run the exercises with Ledgerline as the subject service. |
|
||||
| Coursera — *Measuring and Managing Reliability* (Google Cloud) | Video course | ~1 week | Phase 7, optional. Self-paced SLI/SLO/error-budget deep dive; audit for free. |
|
||||
| Brendan Gregg — the **USE method**; Tom Wilkie — the **RED method** | Articles | ~20 min each | Phase 8 |
|
||||
| Cloudflare — *How Cloudflare runs Prometheus at scale* | Article | ~20 min | Phase 4. The clearest explanation anywhere of what cardinality actually costs. |
|
||||
| SigNoz — *OpenTelemetry FastAPI Instrumentation*; Uptrace — *OpenTelemetry FastAPI* | Articles | ~20 min each | Phase 2 |
|
||||
| Better Stack — *Filebeat vs Logstash*; SigNoz — *OpenTelemetry vs ELK* | Articles | ~30 min | Phase 9 |
|
||||
|
||||
**Where to run it:** Docker Compose on your laptop for Phases 0–8. Don't put this on
|
||||
your k8s cluster until everything works locally — debugging instrumentation and
|
||||
debugging Kubernetes at the same time is how these projects die. Porting it to the
|
||||
cluster is a good Phase 10+ follow-up, and it's the natural bridge to the DevOps half
|
||||
Babak has planned for you.
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Phase 0 — Build the base project, test-first
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`intro.md`](intro.md) · Next: [`phase-1.md`](phase-1.md)*
|
||||
|
||||
**Status:** Done
|
||||
|
||||
**Message:** Both halves complete and verified against the live stack, not just
|
||||
collected.
|
||||
|
||||
- **0a (test suite):** 81 tests across `tests/` (11 files + a 5-module
|
||||
harness), covering all eleven goals G1–G11 and all 13 named pathologies.
|
||||
Verified genuinely red against the empty repo before any implementation
|
||||
existed; `tests/RED-BASELINE.md` records that run. `tests/README.md`
|
||||
documents the invented request-contract fields, measurement methodology,
|
||||
and known-flaky tests.
|
||||
- **0b (implementation):** all seven services (`gateway`, `ledger`, `fraud`,
|
||||
`sanctions`, `insights`, `notifier`, `audit`), `db/schema.sql` +
|
||||
`optional_indexes.sql`, `scripts/seed.py` + `smoke.py`, the frontend, and
|
||||
`docker-compose.yml`. Seeded: 800 accounts, 250,000 transactions, 45
|
||||
merchant categories, 12,000 sanctions entries, in ~12s. `make smoke` and
|
||||
`make test-contract` (64/64) pass; `make test-baseline` and
|
||||
`make test-pathology` pass except for a handful of tests documented below
|
||||
as load-sensitive rather than broken.
|
||||
- **Two changes to the frozen test contract, both made only after flagging
|
||||
them and getting an explicit go-ahead — not applied unilaterally:**
|
||||
- `test_row_counts_match_manifest_and_targets` loosened the `transactions`
|
||||
row-count check from exact equality to `>=`, since other contract tests
|
||||
in the same `make test-contract` run legitimately insert rows first.
|
||||
- `test_pool_starvation_plateaus_throughput` was **removed outright**.
|
||||
Its baseline assertion (throughput scales ≥2.5x from concurrency 5→25)
|
||||
is incompatible with this same spec's own §3 mandate of one uvicorn
|
||||
worker per service — confirmed by reproducing the identical
|
||||
non-scaling behaviour on a bare `GET /healthz` with zero backend
|
||||
logic. `pool_starvation` itself is still fully implemented
|
||||
(`services/ledger/db.py`'s dual-engine pool selection); only that one
|
||||
differential test was dropped.
|
||||
- **Known-flaky, documented rather than papered over:**
|
||||
`test_clean_baseline_is_healthy`, `test_nightmare_degrades_but_survives`,
|
||||
the baseline half of `test_blocking_event_loop_serializes_concurrent_requests`,
|
||||
and `test_missing_index_forces_seq_scan` compare a fast-path measurement
|
||||
against a slow-path one under real concurrent load, and pass at roughly a
|
||||
60–80% rate on ordinary dev hardware depending on momentary host
|
||||
contention. See "Known-flaky tests" in `tests/README.md`.
|
||||
- **Notable implementation deviations** (none change observable behaviour of
|
||||
the 13 pathologies or the request/response contract): gateway's `GET /`
|
||||
serves the frontend rather than §4's generic info JSON (the other five
|
||||
HTTP services still return that JSON as specified); `insights` splits a
|
||||
cheap PK-only `quick_summary` (embedded in every transaction response)
|
||||
from the real cached/N+1-affected 120-day analytics endpoint, so the
|
||||
missing-index pathology doesn't tax every request; `retry_storm` fires
|
||||
all 4 attempts unconditionally rather than stopping at first success
|
||||
(spec's "no circuit breaker" taken literally — also the only way to reach
|
||||
the required lookup-multiplication ratio given `sanctions_errors`' fixed
|
||||
3% rate); `poison_data` resolves via a Redis hash populated at seed time
|
||||
instead of a live table scan, to avoid the deliberately missing index;
|
||||
uvicorn's access log is disabled fleet-wide (redundant with the spec's
|
||||
own logging design, and was double-counting two log-line-based pathology
|
||||
tests); Postgres `max_connections` raised from 100 to 300 (five
|
||||
services' pool sizes can sum past the default under load).
|
||||
|
||||
**Time-box:** 2–3 evenings. Two agent sessions plus one evening of your own
|
||||
reading.
|
||||
|
||||
**Deliverable:** the working Ledgerline system with no observability, plus a
|
||||
black-box integration suite that pins down both its working state and its
|
||||
deliberate brokenness.
|
||||
|
||||
### Why TDD here, when the app itself doesn't matter
|
||||
|
||||
The microservices are throwaway. The *properties* aren't. Everything from Phase 2
|
||||
onward assumes the system still behaves a specific way: seven real services in
|
||||
the path, async work genuinely off the request path, five hot accounts owning a
|
||||
third of the rows, and thirteen pathologies that actually degrade something. You
|
||||
will be editing all seven services for weeks. Without a suite that fails when one
|
||||
of those properties quietly breaks, you'll spend an evening chasing a trace
|
||||
anomaly that turns out to be a regression you introduced yesterday.
|
||||
|
||||
Writing the tests first also forces the contract to be explicit before anything
|
||||
is built — which is what makes a second agent able to implement against it
|
||||
without guessing.
|
||||
|
||||
### 0a — the test suite
|
||||
|
||||
Agent session one, from `phase-0a-test-suite-spec.md`. That agent writes only
|
||||
`tests/` and must not create so much as a stub under `services/`. Its output
|
||||
should be **red**, with a `tests/RED-BASELINE.md` recording why each test fails
|
||||
right now — a test that passes against an empty repo is a broken test.
|
||||
|
||||
The suite asserts goals, not mechanics. Two representative examples worth
|
||||
understanding before you read the code:
|
||||
|
||||
- *"Stop the `sanctions` container and transactions must fail."* This is the test
|
||||
that catches an implementation quietly collapsing seven services into one
|
||||
process. Nothing else would.
|
||||
- *"With `n_plus_one` on, one insights request issues ≥ 50 SQL statements."*
|
||||
Measured via `pg_stat_statements`, not wall clock — because a query count
|
||||
can't be faked with a `sleep()`, and a latency threshold can.
|
||||
|
||||
That second point is the design principle throughout: **prefer assertions that
|
||||
are hard to satisfy without the real mechanism.** Query counts, `EXPLAIN` plan
|
||||
node types, Redis consumer-group lag, container RSS, and genuine unhandled
|
||||
exception paths all have that property.
|
||||
|
||||
### 0b — the implementation
|
||||
|
||||
Agent session two, from `phase-0-base-project-spec.md`, with `tests/` handed over
|
||||
as a **read-only contract**. That agent may not edit, skip, `xfail` or loosen a
|
||||
single test. If something looks impossible it stops and reports rather than
|
||||
adjusting the target.
|
||||
|
||||
Done when `make test` is green three consecutive runs and
|
||||
`git diff --stat tests/` is empty.
|
||||
|
||||
### Then, yourself
|
||||
|
||||
1. **Read every service's `app.py` end to end.** You need to know this codebase
|
||||
cold, because from Phase 2 onward you're modifying all seven services.
|
||||
2. **Run `make chaos-nightmare`, fire a 200-request burst from the frontend, and
|
||||
then try to figure out what's wrong using only `docker compose logs`.** Give it
|
||||
twenty real minutes. You will fail, and the specific way you fail — no timing,
|
||||
no causality, no cross-service correlation, a `X-Request-Id` that dies at the
|
||||
Redis Stream — is the motivation for everything that follows. Write down what
|
||||
you wished you had. That list is your acceptance criteria for Phase 8.
|
||||
|
||||
**Done when:** `make test` is green three runs in a row, all nine acceptance
|
||||
criteria in the base spec pass, and you have that written list of frustrations.
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Phase 1 — The model, and one signal end to end
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Blocked on nothing — Phase 0 is done and the platform is
|
||||
described above. Waiting to begin.
|
||||
|
||||
**Time-box:** one evening.
|
||||
|
||||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||||
than abstract.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||||
Observability is a property of the system: can you ask a *new* question about
|
||||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||||
attributes), not just pre-aggregated counters.
|
||||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||||
- The **API** is what your code and third-party instrumentation libraries call.
|
||||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||||
library can be instrumented without forcing a dependency on any implementation.
|
||||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||||
Application owners configure the SDK; library authors only touch the API.
|
||||
- The **Collector** is a standalone process that receives, processes and exports
|
||||
telemetry. Optional, but you should use one.
|
||||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||||
that process. Attributes describe the individual span/metric/log.
|
||||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||||
aware of, not learning yet.)
|
||||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||||
seven services. The Collector also does the things you don't want in your app:
|
||||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||||
destinations.
|
||||
|
||||
### Resources
|
||||
|
||||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||||
it — just see what "done" looks like.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||||
`ansible-galaxy collection install -r requirements.yml`, change
|
||||
`grafana_admin_password` (vault it), then
|
||||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||||
a single container with everything pre-wired. Useful for a first trace, but
|
||||
don't build on it — its pipeline differs from your real platform in exactly the
|
||||
ways that matter.)*
|
||||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||||
auto-instrumentation:
|
||||
```
|
||||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||||
opentelemetry-bootstrap -a install
|
||||
```
|
||||
and change its command to
|
||||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||||
with `OTEL_SERVICE_NAME=sanctions`,
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||||
Tempo directly.
|
||||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||||
|
||||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||||
loud what the Resource attributes on it mean and where they came from.
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Phase 10 — Package the delivery
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`summary.md`](summary.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on every prior phase being done — this is the wrap-up,
|
||||
not independent work.
|
||||
|
||||
**Time-box:** one evening.
|
||||
|
||||
Babak gave you a task; deliver it like an engineer, not like a student.
|
||||
|
||||
1. **`README.md`** — architecture diagram, one-command start, what each container is
|
||||
for, and a "here's the demo" script.
|
||||
2. **The five-minute demo path**, rehearsed:
|
||||
- Delete every dashboard in the Grafana UI, then `ansible-playbook
|
||||
playbooks/grafana-stack.yml` — the whole platform and every dashboard rebuilt
|
||||
from code, one command.
|
||||
- Submit one transaction from the frontend, then follow it through Grafana: RED
|
||||
dashboard → latency spike → exemplar → trace → span → logs for that span → back
|
||||
to the trace. One request, all three signals, no searching.
|
||||
- `make chaos-nightmare && make load-stress` — watch the SLO burn, the alert fire,
|
||||
and the service map turn red at a specific node.
|
||||
- Show `docs/INVESTIGATION.md` and how you found the pathologies.
|
||||
3. **The written answers** Babak asked for, as documents in the repo, not as a
|
||||
conversation: `docs/SLO.md`, `docs/PIPELINE-CHOICES.md`,
|
||||
`docs/INVESTIGATION.md`, plus a short `docs/CONCEPTS.md` covering P95/P99,
|
||||
RTO/RPO/SLA/SLO/error budget in your own words.
|
||||
4. **The test suite as part of the story.** Show `make test` green, then show the
|
||||
three generations of the call-counting assertion side by side —
|
||||
`docker logs | grep`, then LogQL, then TraceQL — as concrete evidence of what
|
||||
each signal bought you. Most people demo dashboards; almost nobody can show
|
||||
their observability work paying off inside their own test suite.
|
||||
5. **A "what I'd do differently in production" section.** Tail sampling instead of
|
||||
100%; Mimir instead of raw Prometheus for retention; Alloy as a DaemonSet;
|
||||
log volume budgeting; PII redaction in the Collector; multi-tenancy. Naming what
|
||||
you *didn't* do, and why, is the strongest signal of seniority in the whole
|
||||
delivery.
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Phase 2 — Distributed tracing
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-1.md`](phase-1.md) · Next: [`phase-3.md`](phase-3.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 1's `common/telemetry.py` bootstrap existing first.
|
||||
|
||||
**Time-box:** 2–3 evenings. This is the biggest phase and the heart of the task.
|
||||
|
||||
**Why traces before logs and metrics?** Two reasons. First, the SDK bootstrap
|
||||
(Resource, providers, exporters, Collector wiring) is shared across all three
|
||||
signals — do it once here and logs and metrics become incremental. Second, tracing
|
||||
is what Babak actually asked for ("trace a single request, see where it stuck"),
|
||||
and once trace IDs exist, log correlation in Phase 3 works on the first try instead
|
||||
of needing rework.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Trace / span / span context.** A trace is one request's journey; a span is one
|
||||
timed operation within it. Each span carries a trace ID, its own span ID, and its
|
||||
parent's span ID. That parent pointer is the entire data structure.
|
||||
- **Span kinds:** `SERVER` (you handled an inbound request), `CLIENT` (you made an
|
||||
outbound call), `PRODUCER` / `CONSUMER` (async messaging), `INTERNAL`. Getting
|
||||
these right is what makes the service graph render correctly.
|
||||
- **Context propagation.**
|
||||
- *In-process*: OTel uses Python `contextvars`. This works across `await` — but
|
||||
**not** across a bare `threading.Thread` or a `ProcessPoolExecutor` unless you
|
||||
carry the context explicitly.
|
||||
- *Cross-process*: the **W3C Trace Context** headers, `traceparent` and
|
||||
`tracestate`. (B3 is the legacy Zipkin format; you'll see it in older systems.)
|
||||
- The classic bug: any boundary that doesn't carry those headers **starts a brand
|
||||
new trace**. Your Redis Stream hop is exactly this. You'll fix it by hand with
|
||||
`TraceContextTextMapPropagator().inject(...)` into the message payload on the
|
||||
producer and `.extract(...)` on the consumer, then starting the consumer span
|
||||
with that context as parent. This is the single most valuable exercise in the
|
||||
phase — most people never do it manually and never really understand it.
|
||||
- Related: `links` (for a consumer span that batches messages from many traces)
|
||||
and **Baggage** (arbitrary key/values propagated alongside trace context —
|
||||
useful, and a foot-gun, because it's sent on every hop and is visible to
|
||||
downstream systems; never put secrets in it).
|
||||
- **Auto- vs manual instrumentation.** Auto gets you the server span, the HTTP
|
||||
client spans, the DB query spans — 80% of the value for 20 lines of setup. Manual
|
||||
gets you the business semantics: `score_transaction`, `check_sanctions_list`,
|
||||
`settle_ledger_entry`. You need both. Auto-instrumentation alone produces traces
|
||||
that show *which service* was slow but not *why*.
|
||||
- **Span attributes vs metric labels.** Spans are the right home for
|
||||
high-cardinality data (account IDs, transaction IDs, merchant names). Costs scale
|
||||
with volume, not with distinct values. This is why "put it on the span, not the
|
||||
metric" is the standing rule.
|
||||
- **Errors on spans:** set span status to `ERROR` and call `record_exception(e)`.
|
||||
That's how the exception ends up attached to the trace and how the `spanmetrics`
|
||||
connector in Phase 4 will count your error rate.
|
||||
- **Sampling — the concept now, the config later.**
|
||||
- *Head-based*: decide at trace start. Cheap and predictable, but blind — it drops
|
||||
errors at the same rate as successes.
|
||||
- *Tail-based*: decide after the trace finishes. Can keep 100% of errors and slow
|
||||
traces, but must run in the Collector and buffer spans in memory.
|
||||
- For a local demo: sample 100% (`ParentBased(AlwaysOn)`). Understand the trade-off
|
||||
now; you'll configure tail sampling in the Collector in Phase 5.
|
||||
- Whatever you choose, the sampling decision must **propagate** — the sampled flag
|
||||
rides in `traceparent`. Use `ParentBased` samplers so a service doesn't
|
||||
unilaterally drop half of a trace its caller decided to keep.
|
||||
|
||||
### Resources
|
||||
|
||||
- Jessica Garson — *Introduction to OpenTelemetry with Python* (PyTexas 2025, ~30
|
||||
min, free) and *…with Django* (DjangoCon US 2024, ~30 min, free). Watch both.
|
||||
- Your chosen Udemy backbone: the instrumentation + context-propagation sections.
|
||||
- SigNoz — *OpenTelemetry FastAPI Instrumentation* (free article) — the most
|
||||
practical FastAPI-specific setup, including batch/queue env vars.
|
||||
- OpenTelemetry docs → Python → *Instrumentation* and *Propagation*. Reference, not
|
||||
reading.
|
||||
- **OpenTelemetry Demo**: open `src/recommendation` (Python) and the Go/TS services
|
||||
side by side. Look specifically at how they mix auto-instrumentation with
|
||||
hand-rolled business spans, and how the Kafka producer/consumer propagate context.
|
||||
This is the "professional implementation" you asked to study.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Shared bootstrap.** Create `common/telemetry.py` with a single
|
||||
`setup_telemetry(service_name)` that builds the `Resource`, `TracerProvider`,
|
||||
`BatchSpanProcessor` and OTLP exporter, and installs the propagator. Every
|
||||
service calls it once at startup. Keep it under 60 lines.
|
||||
2. **Auto-instrument all seven services**: `FastAPIInstrumentor`,
|
||||
`HTTPXClientInstrumentor`, `SQLAlchemyInstrumentor`, `RedisInstrumentor`. Verify
|
||||
you now get a connected trace across `gateway → ledger → fraud → sanctions`.
|
||||
3. **Manual spans for business logic.** At minimum: `fraud.score_transaction`,
|
||||
`sanctions.lookup_watchlist`, `insights.aggregate_120d`,
|
||||
`ledger.settle`. Add semantic attributes (`db.system`, `http.route`) and
|
||||
namespaced business ones (`ledgerline.account.id`, `ledgerline.risk.score`,
|
||||
`ledgerline.tx.amount_minor`).
|
||||
4. **Propagate across the Redis Stream by hand.** Producer: create a `PRODUCER` span,
|
||||
inject trace context into the message fields. Consumers (`notifier`, `audit`):
|
||||
extract, and start a `CONSUMER` span with the extracted context as parent.
|
||||
5. **Record exceptions** everywhere you currently `logger.exception(...)`.
|
||||
6. Delete the manual `X-Request-Id` plumbing. Trace ID replaces it and does strictly
|
||||
more.
|
||||
7. **Delete `tests/test_no_observability.py`.** It has done its job — it guarded
|
||||
the "before" state so you could measure against it, and from here it's
|
||||
actively wrong. Everything else in `tests/` must still pass: instrumenting
|
||||
the system is not allowed to change its behaviour, and if `make test` goes red
|
||||
after adding spans, you've broken something real. That is exactly the safety
|
||||
net you built it for.
|
||||
|
||||
### Done when
|
||||
|
||||
- One request produces **one connected trace** covering all seven services,
|
||||
including both async consumers.
|
||||
- With `chaos-nightmare` on, you can open a slow trace and *point at the span* that
|
||||
ate the time.
|
||||
- You can explain why `blocking_event_loop` looks different in a trace than
|
||||
`slow_sanctions` does. (Hint: one shows a long span, the other shows a *gap*
|
||||
before spans start, plus every concurrent request in that process degrading
|
||||
together.)
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Phase 3 — Structured logs, correlated to traces
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-2.md`](phase-2.md) · Next: [`phase-4.md`](phase-4.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 2 (trace IDs must exist before log correlation
|
||||
means anything).
|
||||
|
||||
**Time-box:** 1–2 evenings. The easiest phase, and the biggest quality-of-life win.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Structured logging.** Replace prose (`f"processing tx for account {id}"`) with
|
||||
events: a stable event name plus typed fields (`event="transaction_settled"`,
|
||||
`account_id=…`, `amount_minor=…`, `duration_ms=…`). Past-tense event names.
|
||||
Consistent field names across all seven services — `account_id` everywhere, never
|
||||
`accountId` in one place and `acct` in another. This is what makes LogQL queries
|
||||
possible at all.
|
||||
- **Trace–log correlation.** Every log line emitted inside an active span must carry
|
||||
`trace_id` and `span_id`. This is what turns "I have logs and I have traces" into
|
||||
"one pane of glass": click a span → see its logs; click a log line → open its
|
||||
trace. Both directions are configured, and you'll set both up.
|
||||
- **Two ways to ship logs, and which one your platform actually does:**
|
||||
- **stdout → collector scraping → Loki.** The container-native way. Your app just
|
||||
prints JSON; the pipeline handles the rest. Decoupled, survives app crashes,
|
||||
standard in Kubernetes. **This is the only path wired in your Alloy config**
|
||||
(`loki.source.docker` reading the Docker socket).
|
||||
- **OTLP log exporter → collector → Loki.** Logs travel the same path as traces,
|
||||
with the same Resource attributes attached automatically. Cleaner correlation,
|
||||
but the app owns delivery. **On your platform this currently goes nowhere** —
|
||||
Alloy's OTLP receiver has no `logs` output, so OTLP log records are accepted and
|
||||
dropped in silence.
|
||||
- Do both anyway: use stdout for most services, and *extend Alloy* to wire the
|
||||
OTLP logs path for one or two. Then argue the trade-off from experience. That
|
||||
comparison — including the fact that a misconfigured pipeline fails silently
|
||||
rather than loudly — is precisely what Babak will probe.
|
||||
- **The field name is not a style choice here.** Your Loki datasource is provisioned
|
||||
with `matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'`, which is case-sensitive.
|
||||
A log line must carry lowercase `trace_id`. OpenTelemetry's logging
|
||||
auto-instrumentation emits `otelTraceID`, which **will not match** — you'd get
|
||||
logs and traces both arriving, no errors anywhere, and no clickable link. Hence
|
||||
structlog with an explicit processor rather than the auto-injection shortcut.
|
||||
- **Log levels in production.** Run INFO. One service left on DEBUG can out-produce
|
||||
your entire fleet, and Loki bills/indexes on volume.
|
||||
- **What never goes in a log:** credentials, tokens, full card numbers, PII. A naive
|
||||
`json.dumps(request.__dict__)` will happily dump an `Authorization` header. Build
|
||||
redaction into the processor chain, not into each call site.
|
||||
- **Loki's label model — the thing that surprises everyone.** Loki indexes *labels*,
|
||||
not log content. Labels should be few and low-cardinality (`service`, `env`,
|
||||
`level`). Everything else lives in the log line body and is filtered at query time.
|
||||
Putting `trace_id` in a Loki *label* is a classic cardinality catastrophe; put it
|
||||
in the JSON body and use a **derived field** to make it clickable.
|
||||
|
||||
### Resources
|
||||
|
||||
- *Structured Logging in Python With Structlog: Correlating Logs, Traces, and
|
||||
Errors* (dev.to) — free, and the most copy-pasteable structlog + OTel processor
|
||||
setup, including the hex formatting for trace/span IDs.
|
||||
- OpenTelemetry Python Contrib docs → *Logging Instrumentation* — for
|
||||
`OTEL_PYTHON_LOG_CORRELATION` and the auto-injected `otelTraceID` / `otelSpanID`
|
||||
fields, if you'd rather not hand-roll a processor.
|
||||
- Udemy backbone: the Loki + LogQL sections.
|
||||
- Grafana docs → Loki → *Send data via OTLP* and *derived fields*.
|
||||
|
||||
### Build
|
||||
|
||||
1. Replace `common/logging_config.py` with `structlog`, JSON renderer, plus a
|
||||
processor that pulls the current span context and injects `trace_id` / `span_id`.
|
||||
2. Rewrite every log call as an event with fields. Keep it to ~2–4 log lines per
|
||||
handler: request received (or rely on the span), business decision, error.
|
||||
3. Add a redaction processor with a denylist of field names.
|
||||
4. Ship all seven services' logs via **stdout**, and confirm they arrive in Loki
|
||||
with a `container` label (Alloy's relabel rule sets it from the Docker container
|
||||
name). Verify the `trace_id` field is present *and lowercase* in the JSON body.
|
||||
5. **Prove the silent-drop for yourself.** Add an OTLP log exporter to one service,
|
||||
send some records, and watch nothing appear in Loki. Then check the Alloy UI:
|
||||
the receiver shows traffic, and the pipeline shows no logs output. Now extend
|
||||
`config.alloy.j2` — add a `logs` output on the receiver plus an
|
||||
`otelcol.exporter.loki` → `loki.write` chain — re-run the playbook, and watch
|
||||
them arrive. Write down what you'd have done to debug this if you *hadn't* known.
|
||||
That is the most transferable half-hour in the whole project.
|
||||
6. Confirm the provisioned correlation works in both directions: Loki's
|
||||
`derivedFields` on `trace_id` → Tempo, and Tempo's `tracesToLogsV2`
|
||||
(`filterByTraceID: true`) → Loki. Both are already in `datasources.yml.j2`; you
|
||||
are verifying, not writing. If a link doesn't appear, the regex didn't match
|
||||
your field name — check that before anything else.
|
||||
6. **Migrate the test suite's log-counting helpers.** Three pathology tests
|
||||
(`retry_storm`, `no_batching`, `chatty_health`) currently count matching lines
|
||||
in `docker compose logs` — the only cross-service call counter that existed at
|
||||
Phase 0. Rewrite `tests/harness/containers.py::count_log_lines` to query Loki
|
||||
with LogQL over structured fields instead. This is a small change and a
|
||||
pointed one: notice how much shorter and more precise the assertion gets, and
|
||||
that it now works without shelling out to Docker at all. That delta is the
|
||||
value of structured logging, measured on your own code.
|
||||
|
||||
**Done when:** from a slow trace you can click into that request's logs, and from
|
||||
any log line you can click into its trace — in both directions, in two clicks.
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
# Phase 4 — Metrics
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-3.md`](phase-3.md) · Next: [`phase-5.md`](phase-5.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Not blocked technically (metrics don't strictly need tracing
|
||||
first), but sequenced after Phase 3 per the roadmap's ordering.
|
||||
|
||||
**Time-box:** 2 evenings.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Instrument types.**
|
||||
- **Counter** — monotonic, only increases. Requests, errors, messages consumed.
|
||||
- **UpDownCounter / Gauge** — can go both ways. Queue depth, in-flight requests,
|
||||
connection-pool checkouts.
|
||||
- **Histogram** — a distribution. Request latency, payload size. This is what
|
||||
gives you percentiles.
|
||||
- Synchronous instruments are recorded inline; **observable/asynchronous** ones are
|
||||
read via a callback on collection (right for things you *poll*, like pool size).
|
||||
- Avoid Prometheus **Summaries**: they compute quantiles client-side and therefore
|
||||
**cannot be aggregated across instances**. `avg(p99_from_summary)` is arithmetic
|
||||
nonsense. Use histograms.
|
||||
- **Cardinality — the one thing that will bite you.** Every unique combination of
|
||||
metric name + label values is a separate time series in memory. Add `account_id`
|
||||
as a label on a 800-account demo and you've made 800 series; do it in production
|
||||
and you make millions and OOM Prometheus. Rule of thumb: **if a label could exceed
|
||||
~10 000 distinct values in production, it does not belong on a metric.** Put it on
|
||||
a span. Histograms multiply this (one series per bucket per label combination), so
|
||||
don't over-bucket either.
|
||||
- Bounded and fine: `service`, `http.route` (the *template*, `/accounts/{id}`, not
|
||||
the concrete path), `http.status_code` class, `kind`, `currency`.
|
||||
- Unbounded and forbidden: `account_id`, `transaction_id`, `merchant_name`,
|
||||
raw URL, timestamp, error message.
|
||||
- **RED and USE.**
|
||||
- **RED** (Rate, Errors, Duration) — for *request-driven services*. This is your
|
||||
per-service dashboard.
|
||||
- **USE** (Utilization, Saturation, Errors) — for *resources*: CPU, memory,
|
||||
connection pools, queues, disk. Saturation is the one everyone forgets and the
|
||||
one that predicts collapse.
|
||||
- You want both: RED tells you the service is slow, USE tells you which resource
|
||||
is why.
|
||||
- **Span metrics — generate metrics from traces.** Two architectures exist for this,
|
||||
and knowing both is the point:
|
||||
- *In the collector:* the OpenTelemetry Collector's `spanmetrics` and
|
||||
`servicegraph` connectors read spans in flight and emit RED metrics and edge
|
||||
metrics. This is what most blog posts (including the OneUptime article) show.
|
||||
- *In the trace store:* Tempo's **metrics-generator** does the same job after
|
||||
ingestion, remote-writing the result to Prometheus. **This is what your platform
|
||||
does** — `processors: [service-graphs, span-metrics, local-blocks]` with
|
||||
`send_exemplars: true`, remote-writing to a Prometheus started with
|
||||
`--web.enable-remote-write-receiver`.
|
||||
- So RED metrics, the service map and exemplars **already exist** before you write
|
||||
a single metric line. Do not add the collector connectors as well — you'd
|
||||
duplicate every series under a second set of names. **Go find the real metric
|
||||
names in Prometheus** (`/api/v1/label/__name__/values`) before writing PromQL;
|
||||
they will not match the article's.
|
||||
- The trade-off worth being able to argue: collector-side generation is cheaper
|
||||
and works before storage, but sees only what passes through that collector;
|
||||
Tempo-side generation sees everything that landed, survives collector restarts,
|
||||
and can be enabled per-tenant — at the cost of load on the trace store.
|
||||
- **Exemplars.** A histogram bucket can carry a sample trace ID. That's what lets you
|
||||
click a spike on a latency graph and land in the exact trace. On your platform
|
||||
this is already wired end to end: Tempo's generator sets `send_exemplars: true`,
|
||||
and the Prometheus datasource declares `exemplarTraceIdDestinations` with
|
||||
`name: trace_id`. If you emit your own histograms with exemplars, the label must
|
||||
be `trace_id` to match.
|
||||
- **Push vs pull.** Prometheus scrapes (`pull`); OTLP pushes. With the Collector you
|
||||
can do either — receive OTLP from apps, then either expose a `/metrics` endpoint
|
||||
for Prometheus to scrape, or remote-write into Prometheus/Mimir. Understand that
|
||||
the pull model is why Prometheus has service discovery and why `up` exists.
|
||||
|
||||
### Resources
|
||||
|
||||
- Cloudflare — *How Cloudflare runs Prometheus at scale* (free article, ~20 min).
|
||||
Read the first half. Best cardinality explanation available.
|
||||
- Udemy backbone: Prometheus + PromQL sections. Do the exercises; PromQL is a skill,
|
||||
not a fact.
|
||||
- **The OneUptime APM article** — now it's time. Take its `spanmetrics` /
|
||||
`servicegraph` connector config and its RED PromQL queries almost verbatim.
|
||||
- OpenTelemetry docs → Python → *Metrics*; Semantic Conventions → Metrics.
|
||||
|
||||
### Build
|
||||
|
||||
1. Extend `common/telemetry.py` with `MeterProvider` + OTLP metric exporter.
|
||||
2. Custom application metrics, deliberately few:
|
||||
- `ledgerline.transactions.total` (Counter; labels: `kind`, `status`, `currency`)
|
||||
- `ledgerline.transaction.duration` (Histogram; labels: `kind`)
|
||||
- `ledgerline.fraud.score` (Histogram; no labels)
|
||||
- `ledgerline.stream.lag` (Observable Gauge; label: `consumer_group`)
|
||||
- `ledgerline.db.pool.in_use` (Observable UpDownCounter; label: `service`)
|
||||
3. **Get your custom metrics somewhere they'll actually land.** Alloy drops OTLP
|
||||
metrics, so pick a path and implement it:
|
||||
- *Platform grain:* expose `/metrics` on each service and add them to
|
||||
`prometheus_extra_scrape_jobs` in `group_vars/all.yml`, then re-run the
|
||||
playbook. Prometheus scrapes your services directly. Note the irony and the
|
||||
lesson — you're back to a pull model and a second client library, which is
|
||||
exactly the fragmentation OTel exists to fix.
|
||||
- *Match the diagram:* add `prometheus.scrape` + `prometheus.remote_write` to
|
||||
Alloy so app metrics flow through the collector as the architecture drawing
|
||||
intends. This also closes a real diagram-vs-code gap in the repo.
|
||||
- *Go all-OTLP:* add a `metrics` output on `otelcol.receiver.otlp`, wire
|
||||
`otelcol.exporter.prometheus` → `prometheus.remote_write`. One OTLP path for
|
||||
all three signals.
|
||||
Do the first to get moving; do one of the others afterwards and write up which
|
||||
you'd choose for a real platform team and why. That write-up is a Phase 9
|
||||
artefact.
|
||||
4. Deliberately break it: add `account_id` as a label to
|
||||
`ledgerline.transactions.total`, run a 200-request burst, and watch the series
|
||||
count in Prometheus (`count({__name__=~"ledgerline.*"})`). Then remove it. Now
|
||||
you'll never forget.
|
||||
5. Compare your hand-written duration histogram against the `spanmetrics`-generated
|
||||
one. Understand why they differ slightly and when you'd want each.
|
||||
6. **Second migration of the call-counting assertions.** The three tests you moved
|
||||
to LogQL in Phase 3 can now count **client spans** in Tempo instead — a
|
||||
TraceQL query for `sanctions` client spans within a single trace answers
|
||||
"how many lookups did this one transaction make?" directly, which is the
|
||||
question the test was always really asking. Do it and compare all three
|
||||
generations: `docker logs | grep` → LogQL → TraceQL. Keep the diff; it's the
|
||||
clearest possible demonstration of what each signal is actually for, on a
|
||||
problem you already understand.
|
||||
|
||||
**Done when:** you have a RED view per service, a working service map, and you can
|
||||
click a latency spike through to a trace via an exemplar.
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# Phase 5 — Dashboards and alerts as code
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-4.md`](phase-4.md) · Next: [`phase-6.md`](phase-6.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phases 2–4 producing traces, logs and metrics worth
|
||||
dashboarding.
|
||||
|
||||
**Time-box:** 2 evenings. This is an explicit requirement from Babak: *destroy and
|
||||
recreate in seconds.*
|
||||
|
||||
### Concepts
|
||||
|
||||
- Three levels of "as code", and you should know when each applies:
|
||||
1. **File provisioning** — YAML in `provisioning/datasources/` and
|
||||
`provisioning/dashboards/`, plus dashboard JSON files mounted into the
|
||||
container. Grafana loads them at startup. Simplest, no state, no API tokens,
|
||||
perfect for a Compose demo. **This alone satisfies the requirement.**
|
||||
2. **Grafana Foundation SDK** — typed libraries (Go, **Python**, TypeScript, Java,
|
||||
PHP) that *generate* the dashboard JSON programmatically. Now a dashboard is a
|
||||
function: `red_dashboard("ledger")`, called seven times in a loop. Reviewable
|
||||
diffs, reusable panels, no 2000-line JSON blobs. Grafana now points here instead
|
||||
of Grafonnet (which is legacy and unofficial).
|
||||
3. **Terraform provider / Grafana Operator / Git Sync** — for state-managed,
|
||||
multi-environment, or GitOps setups. Overkill here; the right answer *later*,
|
||||
when this lands on your Kubernetes cluster.
|
||||
- **Dashboards should answer questions, not display data.** A good service dashboard
|
||||
answers, in order: is it up, is it serving, is it erroring, is it slow, and where
|
||||
is the pressure. Panels that nobody looks at during an incident are cost, not value.
|
||||
- **Alerts as code too.** Grafana alert rules, contact points and notification
|
||||
policies all provision from YAML. And keep the *alert definition* next to the
|
||||
dashboard that explains it.
|
||||
|
||||
### Resources
|
||||
|
||||
- Grafana docs → *Observability as code* (the index page — it lays out Foundation
|
||||
SDK, file provisioning, Git Sync and Terraform, and tells you which to use when).
|
||||
- Grafana Foundation SDK repo → the Python examples, especially the `red-method`
|
||||
example.
|
||||
- Udemy backbone: dashboard design sections (panel types, transformations,
|
||||
variables, dashboard best practice).
|
||||
|
||||
### Build
|
||||
|
||||
Your platform provisions **datasources only** — there is no dashboards provider at
|
||||
all. So this phase is entirely yours, and the right home for it is inside the
|
||||
Ansible role, using the same mechanism as everything else.
|
||||
|
||||
1. **Extend the role.** Add
|
||||
`roles/observability/templates/grafana-stack/config/grafana/provisioning/dashboards/dashboards.yml.j2`
|
||||
(a file-based dashboard provider pointing at a mounted directory), a
|
||||
`config/grafana/dashboards/` directory for the JSON, and the tasks to template
|
||||
and copy them. The Compose template already mounts
|
||||
`./config/grafana/provisioning` read-only, so the provider is picked up on
|
||||
restart.
|
||||
2. **Generate the JSON from code**, don't hand-write it. Use the **Grafana
|
||||
Foundation SDK (Python)**: one function producing a RED dashboard for a given
|
||||
service name, looped over the seven services; a second producing a system
|
||||
overview (service graph, stream lag, pool saturation, error-budget burn). Commit
|
||||
the generator *and* its output; the Ansible role ships the output.
|
||||
3. **The demo is now `ansible-playbook`.** Delete every dashboard through the
|
||||
Grafana UI, re-run `playbooks/grafana-stack.yml`, and watch them all come back.
|
||||
That is a strictly stronger answer to Babak's "destroy and recreate in seconds"
|
||||
than a local Makefile target, because it's the same command that builds the
|
||||
platform.
|
||||
4. While you're in there, review the provisioned datasources and add what's missing:
|
||||
`tracesToMetrics` on Tempo (pointing at the Tempo-generated span metrics) is a
|
||||
worthwhile addition, and the existing `tracesToLogsV2` has no `tags` mapping —
|
||||
understand why `filterByTraceID: true` alone is sufficient here, and when it
|
||||
wouldn't be.
|
||||
5. Add alert rules — see Phase 7 for the Alertmanager-vs-Grafana-alerting choice.
|
||||
|
||||
**Done when:** re-running the playbook restores every dashboard from scratch, and no
|
||||
dashboard has ever been saved through the Grafana UI.
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# Phase 6 — Load testing with k6, and percentiles
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-5.md`](phase-5.md) · Next: [`phase-7.md`](phase-7.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 4 (metrics to plot k6's output against) and
|
||||
benefits from Phase 5's dashboards existing.
|
||||
|
||||
**Time-box:** 1–2 evenings.
|
||||
|
||||
### Concepts — P95 and P99 properly
|
||||
|
||||
You asked about these specifically, and there's more to them than "the 95th
|
||||
percentile."
|
||||
|
||||
- **A percentile is meaningless without a population and a window.** "p95 = 400 ms"
|
||||
is not a fact; "p95 of `POST /api/transactions` over the last 5 minutes = 400 ms"
|
||||
is.
|
||||
- **You cannot average percentiles.** Not across instances, not across time buckets.
|
||||
`avg(p99)` is wrong and produces a number that looks plausible. The correct move is
|
||||
to aggregate the *histogram buckets* and compute the quantile from the sum:
|
||||
`histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))`.
|
||||
- **Prometheus classic histograms give you an interpolated estimate**, bounded by
|
||||
your bucket edges. If your buckets jump 0.5 s → 1 s and everything lands in that
|
||||
bucket, your "p99" is a linear guess inside a 500 ms range. Choose buckets that
|
||||
bracket your actual SLO threshold. (Native histograms largely solve this and are
|
||||
worth knowing about.)
|
||||
- **Which population?** An overall p99 across all endpoints is dominated by whichever
|
||||
endpoint has the most traffic. Slice by route, or you'll miss that
|
||||
`/api/accounts/{id}/insights` is catastrophic while the aggregate looks fine.
|
||||
- **Tail latency amplification** — the most important idea here for a microservice
|
||||
system. If one user request fans out to 10 backend calls, each with an independent
|
||||
p99 of 1 s, then the chance the *user request* touches at least one slow call is
|
||||
`1 − 0.99¹⁰ ≈ 9.6%`. Your user-facing p99 is worse than any individual service's
|
||||
p99. This is Dean & Barroso's *The Tail at Scale*, and it's why "each service is
|
||||
fine" and "the product is slow" coexist. Ledgerline's fan-out makes this
|
||||
measurable — go measure it.
|
||||
- **p99 vs p99.9:** the users in the deep tail are often your heaviest, most valuable,
|
||||
most data-rich accounts. Ledgerline's five hot accounts are a stand-in for exactly
|
||||
that.
|
||||
- **Coordinated omission** — a load-generator trap. If your generator waits for a
|
||||
response before sending the next request, then when the system stalls, the
|
||||
generator stalls too and simply *doesn't record* the requests that would have been
|
||||
slow. Your measured latency is systematically optimistic. Fix: use an **open
|
||||
model** — a fixed *arrival rate* independent of response times. In k6 that means
|
||||
the `constant-arrival-rate` / `ramping-arrival-rate` executors rather than
|
||||
`constant-vus`. Knowing this is a genuine senior-level differentiator.
|
||||
|
||||
### Concepts — k6
|
||||
|
||||
- **Test types:** smoke (does it work at all), average-load (normal traffic),
|
||||
stress (past normal, find the knee), spike (sudden surge, test recovery),
|
||||
soak (hours, find leaks and lag drift), breakpoint (ramp until failure).
|
||||
Your `unbounded_cache` and `slow_consumer` pathologies are only visible in a soak.
|
||||
- **Executors:** closed model (`constant-vus`, `ramping-vus`) vs open model
|
||||
(`constant-arrival-rate`, `ramping-arrival-rate`). See coordinated omission above.
|
||||
- **`checks` vs `thresholds`:** checks are assertions that record pass/fail but don't
|
||||
fail the run; **thresholds are SLOs expressed in code and they set the process exit
|
||||
code**, which is what makes k6 a CI gate. `http_req_duration: ['p(95)<500']`.
|
||||
- **Output:** k6 can remote-write its own metrics to Prometheus, so the load test's
|
||||
view and the application's view sit on the same Grafana time axis. That side-by-side
|
||||
is the single most convincing screenshot you can show Babak.
|
||||
- k6 tests are JavaScript/TypeScript, which is fine — you're not learning JS, you're
|
||||
writing 60 lines of it.
|
||||
|
||||
### Resources
|
||||
|
||||
- Grafana k6 docs → *Get started*, *Test types*, *Scenarios & executors*,
|
||||
*Thresholds*. Free, genuinely well written; ~2 h to work through.
|
||||
- Grafana workshop — *Introduction to Performance Testing with Grafana k6* (free,
|
||||
hands-on).
|
||||
- Udemy backbone (Aref Karimi's course) has a k6 section if you took that one.
|
||||
- Grafana docs → *k6 output to Prometheus remote write*.
|
||||
|
||||
### Build
|
||||
|
||||
1. `loadtest/` with four scripts: `smoke.js`, `average.js`, `stress.js`, `soak.js`,
|
||||
sharing a `common.js` for setup and payload generation against random accounts.
|
||||
2. Use **arrival-rate executors** for `average` and `stress`. Write down why.
|
||||
3. Thresholds encoding your intended SLOs: `p(95)<500`, `p(99)<2000`,
|
||||
`http_req_failed rate<0.01`.
|
||||
4. Pipe k6 metrics into Prometheus; add a k6 row to your overview dashboard.
|
||||
5. Run the same profile under `chaos-clean` and `chaos-nightmare`. Save both result
|
||||
sets. This before/after pair is your baseline for Phase 8.
|
||||
6. Optional but impressive: have k6 send W3C trace context headers so load-test
|
||||
requests appear as traces, letting you click from a k6 spike into a specific
|
||||
trace. Check the current k6 docs for the tracing module's status — it has moved
|
||||
between experimental and stable.
|
||||
|
||||
**Done when:** `make load-stress` produces a Grafana view where you can see the
|
||||
arrival rate climb, p95 diverge from p50, error rate turn up, and you can name the
|
||||
service that broke first.
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# Phase 7 — SLI, SLO, SLA, error budgets, RTO, RPO
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-6.md`](phase-6.md) · Next: [`phase-8.md`](phase-8.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phase 6's load-test baseline (the latency SLO should
|
||||
come from measured numbers, not be picked from thin air).
|
||||
|
||||
**Time-box:** 1–2 evenings. Mostly thinking, a little config.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **SLI** — a *measurement* of something users experience, expressed as a ratio of
|
||||
good events to valid events. Not "CPU is at 60%". Rather: *the proportion of
|
||||
`POST /api/transactions` requests that returned 2xx within 500 ms.*
|
||||
- **SLO** — an internal *target* on an SLI over a window. "99.5% of transaction
|
||||
requests succeed within 500 ms, measured over a rolling 28 days."
|
||||
- **SLA** — an external, *contractual* promise, with consequences (refunds, credits).
|
||||
Always **looser** than your SLO, so you have room to miss internally without
|
||||
breaching commercially. If your SLA and SLO are the same number, you've made a
|
||||
mistake.
|
||||
- **Error budget** — `1 − SLO`, expressed as an allowance. A 99.9% SLO gives a 0.1%
|
||||
budget; over 30 days that's about **43 minutes**. The budget is the thing that
|
||||
turns reliability from an argument into arithmetic: if there's budget left, ship
|
||||
features and take risks; if it's spent, the policy says you stop feature work and
|
||||
spend on reliability. That policy — agreed in advance by dev, ops and product — is
|
||||
the whole point.
|
||||
- **Burn rate.** How fast you're consuming budget relative to the window.
|
||||
Burn rate 1 = you'll exactly exhaust the budget at window end. The SRE Workbook's
|
||||
multi-window, multi-burn-rate alerting (fast burn → page; slow burn → ticket)
|
||||
is what stops you from either paging on every blip or noticing the slow bleed a
|
||||
month late.
|
||||
- **RTO / RPO** — these are *disaster recovery* metrics, a different axis from SLOs,
|
||||
and it's worth being precise because interviewers probe exactly here.
|
||||
- **RTO (Recovery Time Objective)** — the maximum acceptable *downtime*: how long
|
||||
from failure to restored service. It drives your architecture: replication,
|
||||
warm standby, failover automation, runbook quality.
|
||||
- **RPO (Recovery Point Objective)** — the maximum acceptable *data loss*, measured
|
||||
in time: how far back can you afford to lose. It drives your **backup and
|
||||
replication frequency**. RPO of 15 minutes means backups/WAL shipping at least
|
||||
every 15 minutes.
|
||||
- The relationship worth stating out loud: **SLO governs steady-state quality; RTO
|
||||
and RPO govern catastrophe.** An SLO says "99.5% of requests are fast." RTO says
|
||||
"if Postgres dies we're back within 30 minutes." RPO says "and we lose at most
|
||||
5 minutes of transactions." Both feed the SLA.
|
||||
- Both are **business decisions, not technical ones** — you derive them by asking
|
||||
what an hour of downtime or an hour of lost data actually costs, then buying the
|
||||
cheapest architecture that meets it. Tightening either is expensive and
|
||||
non-linear.
|
||||
|
||||
### Resources
|
||||
|
||||
- Google SRE Workbook → *Implementing SLOs* (the SLI menu by service type is
|
||||
immediately usable), then *Alerting on SLOs*, then *Error Budget Policy*. Free.
|
||||
- Google SRE — *The Art of SLOs*: slides plus participant and facilitator handbooks,
|
||||
all free PDFs. Run the exercises solo with Ledgerline as the subject.
|
||||
- Optional: Coursera *Measuring and Managing Reliability* (Google Cloud) — auditable
|
||||
free, self-paced, deeper.
|
||||
- For RTO/RPO: Splunk's *RPO vs RTO* explainer is a clean, vendor-neutral-enough
|
||||
overview; pair it with your Postgres backup knowledge.
|
||||
|
||||
### Build
|
||||
|
||||
1. Write `docs/SLO.md`. For Ledgerline define, with justification:
|
||||
- Availability SLI/SLO on `POST /api/transactions`
|
||||
- Latency SLI/SLO on the same (pick the threshold from your Phase 6 baseline, not
|
||||
from thin air)
|
||||
- A freshness/lag SLI on the notifier consumer group
|
||||
- A stated SLA that is deliberately looser than each SLO
|
||||
- An RTO and RPO for the Postgres instance, with the backup cadence they imply
|
||||
- An error budget policy: what happens at 50%, 75% and 100% consumed
|
||||
2. Implement recording rules for the SLIs and **multi-burn-rate alert rules**.
|
||||
Your platform ships **Alertmanager** plus a Prometheus rules directory and the
|
||||
sample rules `HostExporterDown` and `HostHighCPU`, so the grain is
|
||||
*Prometheus rule files → Alertmanager*, templated by the Ansible role, with
|
||||
`alertmanager_webhook_url` in `group_vars` for delivery. Follow the grain — but
|
||||
be able to argue the alternative: Grafana unified alerting can evaluate across
|
||||
datasources (including Loki queries) and lives with your dashboards, whereas
|
||||
Alertmanager is metrics-only but is the ecosystem standard and gives you
|
||||
grouping, deduplication, inhibition and silences for free. Note which you chose
|
||||
and why in `docs/SLO.md`.
|
||||
3. Add an error-budget panel to the overview dashboard: budget remaining, current
|
||||
burn rate, projected exhaustion.
|
||||
4. Prove it: run `chaos-nightmare` under load until you visibly burn budget, and
|
||||
watch the fast-burn alert fire.
|
||||
|
||||
**Done when:** you can explain, in one paragraph each and without notes, the
|
||||
difference between SLI/SLO/SLA, between SLO and RTO, and why RPO is a backup-schedule
|
||||
decision.
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# Phase 8 — Bottleneck hunting (the capstone)
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-7.md`](phase-7.md) · Next: [`phase-9.md`](phase-9.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Depends on Phases 2–7 all being in place (traces, logs, metrics,
|
||||
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
|
||||
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
|
||||
|
||||
**Time-box:** 2 evenings. This is where the whole thing pays off.
|
||||
|
||||
### Concepts — how to actually answer "how do I find bottlenecks?"
|
||||
|
||||
Babak asked how you'd *think* about this, not for a list of tools. Here's the
|
||||
method, roughly in order. The discipline is to go **top-down from the user, not
|
||||
bottom-up from the infrastructure** — bottom-up is how people spend three hours
|
||||
tuning a database that wasn't the problem.
|
||||
|
||||
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
|
||||
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
|
||||
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
|
||||
2. **Localise to a service** using RED + the service graph. Which node's duration
|
||||
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
|
||||
the graph from the leaves inward.
|
||||
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
|
||||
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
|
||||
several slow traces plus one fast trace of the same endpoint as a control.
|
||||
4. **Read the trace for five specific shapes:**
|
||||
- **A long span** — the work itself is slow. Look at its attributes: which query,
|
||||
which host, which parameters.
|
||||
- **A gap** — wall-clock time not covered by any span. This is the interesting
|
||||
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
|
||||
blocked event loop, or a missing instrumentation library. Ledgerline's
|
||||
`pool_starvation` and `blocking_event_loop` both show up as gaps.
|
||||
- **Serialisation** — spans laid end to end that could have been concurrent.
|
||||
Ledgerline's `no_batching` looks exactly like this.
|
||||
- **Repetition** — the same span name 150 times in a row. That's your N+1.
|
||||
- **Retries** — the same client span repeated 3–4 times with errors. That's
|
||||
`retry_storm`, and note that it makes an *overload* worse, never better.
|
||||
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
|
||||
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
|
||||
memory. A resource at 100% utilisation with a growing queue is saturated;
|
||||
latency there is a *symptom*, not the cause.
|
||||
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
|
||||
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
|
||||
concurrent slots. If your pool is 5, you queue — and queueing latency grows
|
||||
non-linearly as you approach saturation. This one formula explains most
|
||||
connection-pool and worker-count incidents you'll ever see.
|
||||
7. **Check for tail amplification** (Phase 6) before blaming any single service.
|
||||
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
|
||||
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
|
||||
over the *same duration*. One variable at a time. Never declare a fix from a
|
||||
dashboard that "looks better."
|
||||
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
|
||||
becomes the bottleneck. Re-measure after every change; stop when you're inside
|
||||
the SLO, not when you run out of ideas.
|
||||
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
|
||||
or lock waits inside Postgres, or which line of Python burned the cycles. That's
|
||||
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
|
||||
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
|
||||
|
||||
### Resources
|
||||
|
||||
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
|
||||
for saturation.
|
||||
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
|
||||
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
|
||||
thing ever written on why microservice latency behaves the way it does.
|
||||
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
|
||||
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
|
||||
these; connect them to the trace's DB spans.
|
||||
|
||||
### Build
|
||||
|
||||
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
|
||||
`docs/PATHOLOGIES.md`.**
|
||||
2. Run `make load-average` for 10 minutes.
|
||||
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
|
||||
like a real incident doc: what you observed, what you hypothesised, what you
|
||||
checked, what it turned out to be. Include screenshots.
|
||||
4. When you think you've found them all, run `make chaos-clean`, enable them one at
|
||||
a time, and confirm each signature.
|
||||
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
|
||||
6. Repeat once with a new random set. The second run is where the method actually
|
||||
becomes yours.
|
||||
|
||||
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
|
||||
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Not technically blocked by earlier phases (it stands on its own
|
||||
as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
|
||||
re-diagnose in Kibana already exists.
|
||||
|
||||
**Time-box:** 1–2 evenings.
|
||||
|
||||
Babak's remaining questions are all *judgment* questions. He isn't testing whether
|
||||
you can recite what Filebeat does; he's testing whether you have a framework for
|
||||
choosing. Here's the framework, then the answers as worked examples.
|
||||
|
||||
### The framework — six axes to reason on
|
||||
|
||||
Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
|
||||
|
||||
1. **Where does the data originate?** Inside your application (you can instrument it)
|
||||
or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
|
||||
touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
|
||||
2. **What shape is it already in?** Structured JSON at the source, or unstructured
|
||||
text needing grok/regex parsing? Parsing cost has to land *somewhere*.
|
||||
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
|
||||
4. **What's the destination, and how many?** One backend forever, or several, or one
|
||||
you expect to change?
|
||||
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
|
||||
on every host) distributes cost and adds resilience but multiplies config surface.
|
||||
Parsing centrally concentrates cost and creates a single point of failure — but
|
||||
one place to change rules.
|
||||
6. **What already exists, and what does it cost?** Existing Elastic investment,
|
||||
existing licences, team skills, and licensing constraints (Elastic's Filebeat and
|
||||
Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
|
||||
Bit and Vector are permissively licensed). Migration is never free.
|
||||
|
||||
### The answers
|
||||
|
||||
**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
|
||||
from a host or container and forwards them with minimal transformation — and your
|
||||
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
|
||||
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
|
||||
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
|
||||
included. Reach for it when the source is a file you don't control the format of, the
|
||||
app can't be modified, and Elasticsearch is where it's going.
|
||||
|
||||
**When to use Logstash.** When you need real *transformation*: parsing unstructured
|
||||
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
|
||||
splitting or merging events, conditional routing to multiple destinations, or
|
||||
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
|
||||
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
|
||||
per host.
|
||||
|
||||
**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
|
||||
architecture and the reason is a **division of labour between edge and centre**:
|
||||
|
||||
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
|
||||
backpressure, ships onward. You want the thing running in a thousand places to be
|
||||
small and boring.
|
||||
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
|
||||
enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
|
||||
you want them in a handful of places you can redeploy, not on a thousand hosts.
|
||||
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
|
||||
persistent queue absorbs it and Filebeat backs off gracefully, rather than every
|
||||
host queuing independently or dropping.
|
||||
|
||||
So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
|
||||
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
|
||||
need central buffering. Skip Logstash when your logs are already structured JSON and
|
||||
Elasticsearch ingest pipelines can do the little parsing that remains.
|
||||
|
||||
**When to use OpenTelemetry.**
|
||||
|
||||
- You own the application code and want telemetry generated *from inside it* — spans
|
||||
with business context, custom metrics, logs already carrying trace IDs.
|
||||
- You need **more than one signal**, correlated. This is the strongest argument: one
|
||||
SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
|
||||
threaded through all three. Neither Filebeat nor Logstash does traces.
|
||||
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
|
||||
data model across all of them.
|
||||
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
|
||||
Jaeger by editing Collector YAML.
|
||||
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
|
||||
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
|
||||
rewriting — without touching the app.
|
||||
|
||||
**When *not* to use OpenTelemetry.** This is the question that separates people who
|
||||
have used it from people who have read about it.
|
||||
|
||||
- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
|
||||
an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
|
||||
file-based collector.
|
||||
- **You only need one signal from one source with no correlation.** A single nginx
|
||||
access log going to a single log store: OTel is more machinery than the job needs.
|
||||
Fluent Bit or Filebeat is simpler and finishes the job today.
|
||||
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
|
||||
Elastic estate with hundreds of Beats modules, dashboards and detection rules is
|
||||
years of investment. "We should use OTel because it's the standard" is not a
|
||||
business case. Note the direction of travel though: the Collector can already
|
||||
receive from and export to most of these, so a *gradual* migration is usually the
|
||||
right shape — often by putting a Collector or Vector in front of the existing
|
||||
agents rather than ripping them out.
|
||||
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
|
||||
where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
|
||||
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
|
||||
a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
|
||||
retention, different query patterns, different cost model.
|
||||
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
|
||||
matured at different rates per language, and profiling is still alpha. Check the
|
||||
status page for your language before betting on a signal.
|
||||
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
|
||||
on it. A two-person team that needs error tracking by Friday should install Sentry.
|
||||
|
||||
The senior framing to give Babak: *these tools aren't competitors so much as
|
||||
different layers.* OpenTelemetry is a **generation and transport standard** for
|
||||
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
|
||||
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
|
||||
Elasticsearch are **stores**. Most real architectures use one from each layer, and
|
||||
the interesting question is always which layer a given problem lives in.
|
||||
|
||||
### Resources
|
||||
|
||||
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
|
||||
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
|
||||
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
|
||||
read these to see how the Collector does the Filebeat/Logstash job itself.
|
||||
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
|
||||
caveats.
|
||||
|
||||
### Build — and this is no longer hypothetical
|
||||
|
||||
Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
|
||||
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
|
||||
Filebeat reading Docker container logs and Logstash parsing JSON into daily
|
||||
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
|
||||
questions — the codebase he handed you contains both worlds.
|
||||
|
||||
1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
|
||||
flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
|
||||
Alloy → Loki. Compare them on the same data.
|
||||
2. Test the direct ingest path the README documents — pipe a JSON line into
|
||||
Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
|
||||
parsing happens before you change it.
|
||||
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
|
||||
that parses them. Then do the same job with Alloy's `loki.process` stages. Time
|
||||
both. Note which one you'd rather debug at 2 a.m.
|
||||
4. Compare the *investigation experience* directly: take one incident you already
|
||||
diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
|
||||
you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
|
||||
wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
|
||||
for some questions, and Loki's cost model genuinely beats Elasticsearch's for
|
||||
others.
|
||||
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
|
||||
licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
|
||||
permissively licensed. For some employers that alone decides it.
|
||||
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
|
||||
valuably — what you actually found annoying about each. Having run both stacks
|
||||
against the same application is the difference between reciting a comparison and
|
||||
arguing one.
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# Roadmap summary — all phases at a glance
|
||||
|
||||
*Part of `docs/roadmap/` — the one-page overview. Read [`intro.md`](intro.md)
|
||||
first for platform/background context; read the individual `phase-N.md` file
|
||||
for the phase you're actually working on. This file exists so an agent (or
|
||||
you) can check overall status and sequencing without loading every phase's
|
||||
full detail.*
|
||||
|
||||
Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase
|
||||
file follows the same shape: **Status**, **Message** (what's done, notes,
|
||||
warnings), **Time-box**, **Concepts**, **Resources**, **Build**, **Done when**.
|
||||
|
||||
## Status and focus
|
||||
|
||||
| Phase | File | Status | Focus | Time-box | Main output |
|
||||
|---|---|---|---|---|---|
|
||||
| 0a | [`phase-0.md`](phase-0.md) | Done | Integration test suite (agent, tests only) | 1 evening | Red suite + `RED-BASELINE.md` |
|
||||
| 0b | [`phase-0.md`](phase-0.md) | Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
|
||||
| 1 | [`phase-1.md`](phase-1.md) | Not started | Concepts + first trace | 1 evening | One trace in Tempo |
|
||||
| 2 | [`phase-2.md`](phase-2.md) | Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
|
||||
| 3 | [`phase-3.md`](phase-3.md) | Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
|
||||
| 4 | [`phase-4.md`](phase-4.md) | Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
|
||||
| 5 | [`phase-5.md`](phase-5.md) | Not started | Dashboards as code | 2 evenings | `make obs-reset` |
|
||||
| 6 | [`phase-6.md`](phase-6.md) | Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
|
||||
| 7 | [`phase-7.md`](phase-7.md) | Not started | SLO / error budget / RTO / RPO | 1–2 evenings | `docs/SLO.md` + burn-rate alerts |
|
||||
| 8 | [`phase-8.md`](phase-8.md) | Not started | Bottleneck hunting | 2 evenings | `docs/INVESTIGATION.md` |
|
||||
| 9 | [`phase-9.md`](phase-9.md) | Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | `docs/PIPELINE-CHOICES.md` |
|
||||
| 10 | [`phase-10.md`](phase-10.md) | Not started | Delivery | 1 evening | README + rehearsed demo |
|
||||
|
||||
## What each phase is, in one line
|
||||
|
||||
- **0 — Build the base project, test-first.** Write the black-box test suite,
|
||||
then implement the seven-service system against it as a frozen contract.
|
||||
**Done** — see `phase-0.md` for the full implementation report.
|
||||
- **1 — The model, and one signal end to end.** Learn the OTel API/SDK/Collector
|
||||
split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via
|
||||
Ansible and get one auto-instrumented trace into Tempo.
|
||||
- **2 — Distributed tracing.** The heart of the task: auto- and
|
||||
manual-instrument all seven services, and hand-propagate trace context
|
||||
across the Redis Stream hop, which carries none by default.
|
||||
- **3 — Structured logs, correlated to traces.** Replace prose logging with
|
||||
structured events carrying a lowercase `trace_id`, so Loki and Tempo link
|
||||
in both directions.
|
||||
- **4 — Metrics.** Instrument RED metrics per service, understand cardinality
|
||||
limits, and route app metrics to Prometheus given Alloy drops OTLP metrics
|
||||
silently. Tempo's metrics-generator already produces span metrics and the
|
||||
service graph — don't duplicate it in the collector.
|
||||
- **5 — Dashboards and alerts as code.** Generate dashboard JSON with the
|
||||
Grafana Foundation SDK and ship it through the Ansible role, so "destroy
|
||||
and recreate in seconds" is a real, repeatable command.
|
||||
- **6 — Load testing with k6, and percentiles.** Learn why percentiles can't
|
||||
be averaged, what coordinated omission is, and drive Ledgerline with
|
||||
arrival-rate executors to get a clean before/after load profile.
|
||||
- **7 — SLI, SLO, SLA, error budgets, RTO, RPO.** Define Ledgerline's SLIs and
|
||||
SLOs from the Phase 6 baseline (not from thin air), write an error-budget
|
||||
policy, and wire multi-burn-rate alerts.
|
||||
- **8 — Bottleneck hunting (the capstone).** Given a random, undisclosed set
|
||||
of chaos flags, find the pathologies from telemetry alone using a top-down,
|
||||
hypothesis-driven method, and write it up as an incident doc.
|
||||
- **9 — Tool-selection judgment.** Reason through Filebeat vs Logstash vs
|
||||
OpenTelemetry on six concrete axes, using the platform's real second stack
|
||||
(Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
|
||||
- **10 — Package the delivery.** README, a rehearsed five-minute demo, the
|
||||
written deliverables (`SLO.md`, `PIPELINE-CHOICES.md`, `INVESTIGATION.md`,
|
||||
`CONCEPTS.md`), and an honest "what I'd do differently in production"
|
||||
section.
|
||||
|
||||
## If you have to cut scope
|
||||
|
||||
Cut Phase 9's build (keep the writing) and Phase 6's soak test first. **Do not
|
||||
cut Phase 8** — it's the only phase that proves you learned anything.
|
||||
Reference in New Issue
Block a user