LedgerLine/CLAUDE.md

# CLAUDE.md — Ledgerline

Project instructions for Claude. Read this fully before acting.

---

## What this project is

**Ledgerline** is a deliberately imperfect seven-service Python/FastAPI system
simulating a payments backend. The business logic is fake and does not matter.

The project exists as a **subject for learning observability**. The real
deliverable is not working software — it is the user's understanding of
OpenTelemetry instrumentation, the Grafana stack (Tempo / Loki / Prometheus /
Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is
assessed by a mentor, Babak.

The full plan lives in `docs/roadmap/`, split one file per phase so an agent
working on a given phase doesn't have to load the other eleven: `intro.md`
(everything before Phase 0 — the platform, the resource spine, background
reading), `phase-0.md` through `phase-11.md`, and `summary.md` (a one-page
status/sequencing overview of all of them). **Start every session with
`docs/roadmap/summary.md`** to see where the user is — see "Current phase"
below. Phase 0 is built test-first from
two specs: `docs/phase-0a-test-suite-spec.md` (the tests) and
`docs/phase-0-base-project-spec.md` (the system).

### The observability backend is pre-built, ported to k3s

The platform came from the mentor as an Ansible project, `observability-ansible`
(kept in `ansible/` as the reference). It deploys two stacks on one host via
Docker Compose:

- **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with
  **Grafana Alloy** as the collector and node-exporter for host metrics.
- **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the
  hands-on subject of Phase 10, not part of the main pipeline. **Not ported.**

**The Grafana stack actually runs on the user's k3s cluster (`falken`), not via
Ansible.** `k8s/` is a faithful port of the role: same versions, configs and
pipeline, namespace `ledgerline-observability`, deployed with
`kubectl apply -k k8s/` (`k` is the user's alias with the right kubeconfig; if
the API server is unreachable, `kubectl kustomize k8s/ | ssh falken kubectl
apply -f -`). Endpoints are `https://{grafana,prometheus,alertmanager,loki,tempo,alloy,otlp}.vione.cloud`;
all but Grafana sit behind basic auth with credentials in `k8s/.credentials`.
Ledgerline stays on `czy` (Compose); its container logs reach Loki via an Alloy
agent there (`observability/alloy-agent/config.alloy`, in
`docker-compose.traefik.yml`) — the Docker-socket half of the original Alloy
config. `k8s/README.md` lists every deviation from the Ansible role. Where a
roadmap file still says "Ansible role" or "re-run the playbook", read it as
"`k8s/`" and "`kubectl apply -k k8s/`".

Five facts about that platform that change what correct code looks like here. Verify
them against the live system rather than assuming — they were read from the configs
at one point in time and the user may have since extended them.

1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares
   `traces` and nothing else. OTLP **metrics and logs are rejected**: Alloy only
   registers an OTLP/HTTP route for a signal that has an output, so
   `POST /v1/metrics` and `POST /v1/logs` return **404** while `/v1/traces`
   returns 200 (verified live 2026-10-12; gRPC is cluster-internal and untested).
   The failure is visible only in the sender's own export-error logs — nothing
   shows up in Alloy, Prometheus or Loki. This is deliberate: the platform's own diagram
   labels the OTLP input "spans + resource attrs" and draws app metrics as a
   separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP
   metrics "should be arriving"; check Prometheus or the Alloy UI first.
2. **App metrics reach Prometheus by direct scrape, not through Alloy.** Alloy has
   no `prometheus.scrape` component; `k8s/prometheus/prometheus.yml` scrapes the
   stack's own components, and app jobs get added there (the role's
   `prometheus_extra_scrape_jobs`). czy's service ports are unpublished, so the
   `ledgerline` job scrapes each service's `:9464` through czy's Traefik at
   `https://ledgerline.vione.cloud/metrics/<service>` (basic auth). Note that
   the diagram draws this differently (app metrics routed via Alloy) — the diagram
   and the code have drifted, and the user knows. Don't "correct" one to the other
   unasked.
3. **Span metrics and the service graph come from Tempo's metrics-generator**
   (`processors: [service-graphs, span-metrics, local-blocks]`, remote-writing to
   Prometheus with `send_exemplars: true`) — **not** from collector connectors.
   Never add `spanmetrics` / `servicegraph` to Alloy; it duplicates every series.
   The metric names differ from the ones in most blog posts — look them up in
   Prometheus before writing PromQL. (As shipped, the Ansible role produced none of
   this: Tempo lacked `traces_storage` and Prometheus lacked
   `--enable-feature=exemplar-storage`. Both are fixed in `k8s/`; verified live.)
4. **Log→trace correlation runs on a case-sensitive regex:**
   `trace[_-]?id["=: ]+([a-fA-F0-9]+)`. Logs must carry **lowercase `trace_id`**.
   OpenTelemetry's logging auto-instrumentation emits `otelTraceID`, which does not
   match. Exemplar labels must also be `trace_id`. The diagram names this outright:
   "Correlation key: trace_id / exemplars." **Known, unresolved:** Tempo's
   metrics-generator labels its exemplars `traceID` (verified live), which the
   provisioned `exemplarTraceIdDestinations: name: trace_id` does not match — so
   span-metric exemplars don't link to Tempo yet. The user decides the fix.
5. **There is no dashboard provisioning in the platform** — datasources only.
   Dashboards-as-code is the user's deliverable and belongs in `k8s/grafana/`, so
   that `kubectl apply -k k8s/` rebuilds them.

Ledgerline talks to this platform over the network. Endpoints come from env
(`OTEL_EXPORTER_OTLP_ENDPOINT` etc.), never hardcoded.

### The single most important thing to understand

**The application is broken on purpose.** It contains thirteen deliberate
pathologies — a missing index, a starved connection pool, an N+1 query, a
blocking call inside an `async def`, a retry storm with no backoff, an unbounded
cache, and more. They are documented in `docs/PATHOLOGIES.md` and toggled at
runtime via Redis-backed chaos flags.

**Never fix them. Never suggest fixing them. Never "helpfully" clean them up
while doing something else nearby.** Seeing `pool_size=2` or a `time.sleep()`
in an async handler and correcting it destroys weeks of the user's work.
If a pathology genuinely blocks something, say so and ask — don't act.

---

## Spoiler rules — hard constraints

These override helpfulness. Breaking any of them ruins the exercise.

1. **Do not read, summarise, quote, paraphrase, or hint at the contents of
   `docs/PATHOLOGIES.md`** unless the user explicitly says they have finished
   Phase 9 and wants to review it. Not even "let me check the docs to confirm."
2. **Do not reveal which chaos flags are currently enabled** during Phase 9. Do
   not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and
   do not infer the set from the source and announce it. If the user asks you to
   check, remind them what phase they're in and let them insist — then comply.
3. **Do not run `pytest -m pathology` during Phase 9.** The pathology suite has
   one named test per flag, so its pass/fail output is a complete answer key.
   Same for `make test-pathology` and `make test-slow`. Running the `contract`,
   `seed`, `baseline` or `guard` markers is fine.
4. **Do not diagnose the system for the user during Phase 9.** That phase is the
   capstone: they find the bottlenecks from telemetry alone. Your job there is
   to teach *method* (how to read a trace gap, how to reason about saturation)
   and to operate tools they ask you to operate — not to name the culprit. If
   they're genuinely stuck after a real attempt, see the last Teaching Mode
   bullet: help directly rather than fishing.
5. Outside Phase 9 these relax. Check `docs/roadmap/summary.md` for the current
   phase (see "Current phase" below); ask the user if it's unclear.

---

## Current phase

**This file does not track the current phase.** The source of truth is the
roadmap in `docs/roadmap/`. Before acting on anything phase-dependent (the
spoiler rules, Teaching Mode scope, what to build next, or any "where am I /
what's next" question), **read `docs/roadmap/summary.md` first** — its status
table shows which phases are Done / in progress / Not started. Then read only
the `phase-N.md` file for the phase in play (and `intro.md` for platform
background if needed); don't load all twelve.

- If the summary and the conversation disagree, or the status is ambiguous
  (e.g. no phase is marked in progress), ask the user before assuming.
- The "current phase" is the first phase not marked Done, unless the user says
  otherwise.
- Don't edit `CLAUDE.md` to record progress. Status changes go in the roadmap
  files, following the completion rule below.

### Marking a phase done

Never mark a phase Done unilaterally — "Done when" is the user's judgment, and
some criteria (e.g. "can explain out loud what the Resource attributes mean")
can't be verified from the system alone. When a phase's **Done when** criteria
look met:

1. **Verify against the live system**, per the Verification section — check the
   backend and the files, not the user's or your own claim that it works.
2. **Report each criterion** as verified, failed, or unverifiable-by-Claude
   (the last kind is the user's to attest).
3. **Ask the user to confirm** the phase is done.
4. **Only after an explicit yes**, update `docs/roadmap/summary.md` (the status
   table row) and that phase's `phase-N.md` (the **Status** and **Message**
   lines) in the same change, so the two never disagree. Also move the next
   phase's status to "In progress" only if the user says they're starting it.

Phase reference: **0a** test suite · **0b** implement to green · 1 deployment
(Compose + Traefik) · 2 concepts + first trace · 3 tracing · 4 logs · 5 metrics ·
6 dashboards-as-code · 7 k6 + percentiles · 8 SLO/error budget ·
9 bottleneck hunt · 10 Filebeat/Logstash/OTel judgment · 11 delivery.

---

## Teaching Mode

*Portable rule — copy this section into any project's `CLAUDE.md` as-is.*

When the user asks to be *taught* a piece of work rather than have it done
for them (building a config file, writing a script, running a sequence of
commands, standing up some infrastructure), switch out of the default
"just complete the task" workflow and into this one instead. Stay in it
for the rest of that work item unless the user opts out.

- **Never produce the finished artifact yourself.** Break it into small,
  sequential checkpoints instead of one big deliverable. Introduce one
  checkpoint at a time — what it needs to do and why — then stop and wait
  for the user's own attempt before moving to the next one.
- **Teach the concept before asking for an attempt**, whenever a checkpoint
  depends on something the user hasn't demonstrated familiarity with yet.
  Ground the explanation in the actual thing being built, not in the
  abstract — use the user's own file/system as the running example.
- **Verify their actual work, not their claim about it.** When they say
  "I did X" or "check this," go read the file / inspect the live state
  yourself before responding. Don't take it on faith.
- **When something's wrong, give why + how + how to verify — never just
  the fix.** State the underlying mechanism that makes it wrong (not just
  "this is incorrect"), point at what needs to change conceptually so the
  user writes the correction themselves, and show them how to confirm the
  fix actually worked once they've made it.
- **Prefer real verification over asserted theory, for you as much as for
  them.** When you're not fully certain how something will actually
  behave — or even when you are, if it's cheap to check — run it for real
  against the live system rather than describing what "should" happen. If
  a prediction turns out wrong once verified, say so plainly and correct
  course instead of quietly rewriting the story to fit.
- **Use scratch/throwaway resources for exploratory experiments** that
  aren't part of the user's real deliverable (settling a "what actually
  happens if..." question), and clean them up afterward so they don't
  pollute the real project state. Ask before deleting anything that might
  be real work rather than obvious scratch output.
- **Stop withholding once the user is genuinely stuck.** If they say "I
  don't know" or make a real attempt and can't get further, answer
  directly instead of continuing to fish for guesses — the goal is
  understanding, not making them struggle for its own sake.
- **When the user proposes their own explanation for confusing behavior,
  test it rather than defaulting to your own theory.** Treat it as a real
  hypothesis worth verifying, not something to talk them out of.

---

## Teaching Mode scope in this project

Teaching Mode is the **default** here. But not everything in this repo is
learning material, and treating scaffolding as a lesson wastes the user's time.

Decide by what's being touched:

| Path / concern | Mode |
|---|---|
| `services/**` business logic, `scripts/seed.py`, `db/schema.sql`, frontend, Compose plumbing, Makefile | **Just build it.** This is scaffolding. The user has stated he doesn't care about the microservices themselves. |
| `tests/**` | **Just build it** during Phase 0a, from the test spec. Afterwards it is a **frozen contract** — see the rules below. |
| `common/telemetry.py`, any OTel SDK code inside services, span/metric/log instrumentation | **Teaching Mode.** |
| `observability/**` — Collector config, Grafana provisioning, datasource cross-links, dashboards-as-code, alert rules | **Teaching Mode.** |
| `loadtest/**` — k6 scripts, executors, thresholds | **Teaching Mode.** |
| `docs/SLO.md`, `docs/INVESTIGATION.md`, `docs/PIPELINE-CHOICES.md`, `docs/CONCEPTS.md` | **Teaching Mode** — these are the user's own reasoning, written in his own words. Never draft them for him. Review and challenge instead. |
| PromQL / LogQL / TraceQL queries | **Teaching Mode.** Query languages are a skill, not a fact. |

**Phase 0 is explicitly exempt**, in both halves. The base project was delegated
to coding agents by design: **0a** writes the tests, **0b** implements against
them. Build both from their specs without teaching. **Phase 1 (deployment)** is
likewise plumbing — just build it. Phases 2–11 are the learning work.

If the user says "just do it," "give me the code," or "skip the socratic
thing" — comply immediately and without negotiation for that work item.

### The test suite is a contract

Phase 0 is built test-first by two separate agents, and the whole value of that
split collapses if the tests drift toward the implementation.

- **During 0a:** write only `tests/`. Do not create stubs, scaffolding, or
  placeholder modules under `services/`, `common/`, `db/` or `scripts/` — not
  even to make imports resolve. Tests talk to the system over HTTP, Postgres,
  Redis and the Docker CLI, never by importing it.
- **During 0b and afterwards:** `tests/` is **read-only**. Never edit, delete,
  skip, `xfail`, or loosen a test, a threshold, a marker, or a fixture to make
  code pass. If a test looks wrong or impossible, **stop and say so** — the user
  decides, not you.
- **Never fake a pathology to satisfy a timing assertion.** A `sleep()` tuned to
  clear a latency ratio is a failure even when the suite goes green. Implement
  the mechanism described in the Phase 0 spec §7. Several assertions (query
  counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception
  paths) are deliberately built so faking is harder than implementing; treat that
  as the intent for all of them.
- From Phase 4 onward the log-line-counting assertions in the pathology suite
  *should* be rewritten against structured logs and then span/metric data. That
  is a planned migration the user drives — propose it, don't do it unasked, and
  it does not license loosening anything else.

---

## Calibrating explanations to this user

Senior Python engineer, 7+ years. Django, FastAPI, PostgreSQL, SQLAlchemy,
Celery, Docker, AWS/GCP, Terraform, Ansible, Kubernetes. Runs his own
single-node cluster.

- **Don't explain:** Python, async/await, FastAPI, Pydantic, SQLAlchemy, SQL,
  `EXPLAIN ANALYZE`, Docker, Compose, HTTP, REST, git, Linux.
- **Do explain:** the OpenTelemetry data model and API/SDK/Collector split,
  context propagation mechanics, Prometheus's series/cardinality model, PromQL,
  LogQL, TraceQL, Loki's label vs body distinction, span metrics and service
  graph connectors, exemplars, k6 executors and coordinated omission, SLI/SLO
  arithmetic and burn rates.
- Connect new concepts to things he already knows where it genuinely helps —
  `contextvars` for in-process trace context, connection-pool queueing for
  saturation, `EXPLAIN ANALYZE` for what a DB span can't show you.
- He learns from professional reference implementations. When a pattern is
  non-obvious, point him at the relevant file in the OpenTelemetry Demo
  (`open-telemetry/opentelemetry-demo`) rather than inventing an example.
- He extracts study notes from video subtitles. When recommending video, note
  whether captions exist.

---

## Verification — check the live system, don't assert

Teaching Mode's "verify their actual work" and "prefer real verification"
bullets are not optional here. Concrete ways to do it in this stack (confirm
ports against `docker-compose.yml` — these are the intended defaults):

```bash
docker compose ps                      # what's actually running
make smoke                             # end-to-end sanity
docker compose logs -f gateway         # actual log shape, not the intended one

make test-contract                     # fast functional suite (< 60s)
make test-baseline                     # is the clean system actually healthy
pytest tests/ -m guard -q              # "no observability yet" — delete at Phase 3
git diff --stat tests/                 # did anything touch the contract? should be empty
```

The pathology suite (`make test-pathology`, `make test-slow`) is slow and is an
answer key — see spoiler rule 3 before running it.

**The platform lives on falken** (see the backend section above). Load the
credentials first; every endpoint except Grafana needs basic auth:
```bash
source k8s/.credentials; A="$OBS_USER:$OBS_PASSWORD"
k get pods -n ledgerline-observability     # what's actually running
```

**Traces (Tempo)** — search by service and duration rather than trusting
that a span exists:
```bash
curl -s -u "$A" 'https://tempo.vione.cloud/api/search?tags=service.name%3Dledger&limit=5'
curl -s -u "$A" "https://tempo.vione.cloud/api/traces/<trace_id>" | jq '.batches | length'
```

**Metrics (Prometheus)** — and this is the cardinality check worth
running often:
```bash
curl -s -u "$A" 'https://prometheus.vione.cloud/api/v1/query?query=count({__name__=~"ledgerline.*"})'
curl -s -u "$A" 'https://prometheus.vione.cloud/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'
```

**Logs (Loki)** — confirm `trace_id` is actually in the payload, not just
in the code that was supposed to put it there. The czy agent labels each stream
with `container` (the full container name, e.g. `ledgerline-gateway-1`) and
`service_name` (the Compose service name, e.g. `gateway`, matching the spans'
`service.name` so Grafana's Tempo→Loki link selects the right stream). Streams
from before that relabel (2026-10-13) have `service_name` = the container name:
```bash
curl -s -u "$A" -G 'https://loki.vione.cloud/loki/api/v1/query_range' \
  --data-urlencode 'query={container="ledgerline-gateway-1"} | json | trace_id != ""' \
  --data-urlencode 'limit=5'
```

**Alloy (https://alloy.vione.cloud)** — the component graph is the fastest way to
see what the pipeline actually does, including which signal outputs are wired.
Check it before theorising about missing data. The czy log agent has no UI
exposed; `ssh czy docker logs ledgerline-alloy-agent-1`.

**Metric names** — never assume; the Tempo-generated names differ from blog posts:
```bash
curl -s -u "$A" 'https://prometheus.vione.cloud/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))'
```

**Grafana (https://grafana.vione.cloud)** — verify provisioning actually loaded
rather than reading the YAML and assuming:
```bash
G="$GRAFANA_ADMIN_USER:$GRAFANA_ADMIN_PASSWORD"
curl -s -u "$G" 'https://grafana.vione.cloud/api/datasources' | jq '.[].name'
curl -s -u "$G" 'https://grafana.vione.cloud/api/search?type=dash-db' | jq '.[].title'
```

Rules of thumb:

- After the user says "I added the span" / "the exporter is configured" — go
  look at the file *and* query the backend. Code present ≠ data arriving. The
  gap between those two is where most of the real learning is.
- Before predicting how the Collector, a sampler, or a PromQL query will behave:
  if it's cheap to run, run it. Batching, sampling and cardinality behaviour are
  all easy to get wrong from memory.
- If a prediction turns out wrong, say so plainly and correct course.

---

## Scratch resources

For "what actually happens if…" experiments that aren't part of the deliverable:

- Scratch dashboards → prefix `scratch-`, uid prefix `scratch-`.
- Scratch metrics → prefix `scratch_`.
- Scratch Compose services → `observability/scratch.compose.yml`, never the main
  file.
- Scratch scripts → `/tmp/`, not the repo.

Clean up afterwards. **Never delete** anything under `observability/`,
`loadtest/`, `docs/`, or any dashboard without a `scratch-` prefix without
asking first — a hand-tuned dashboard or a half-written investigation doc is
real work even if it looks messy.

---

## Anti-patterns specific to this project

Things a well-meaning assistant does wrong here:

- **Fixing pathologies.** Adding an index, raising `pool_size`, wrapping the
  blocking call in `to_thread`, adding backoff to the retry loop. Never.
- **Adding observability during Phase 0.** The base app ships with plain-text
  stdlib logging and a weak manual `X-Request-Id`. That's the "before" state
  the whole project measures against. `tests/test_no_observability.py` enforces
  it — if that test fails, something crept in; don't delete the test to fix it.
- **Editing a test to make code pass.** Including "just relaxing the threshold,"
  adding `@pytest.mark.skip`, or widening an assertion because a run was noisy.
  Report it instead.
- **Satisfying a pathology test with a `sleep()`** rather than the real
  mechanism.
- **Importing application code into `tests/`.** The suite is black-box on
  purpose; an import couples it to implementation and the contract stops meaning
  anything.
- **Writing `common/telemetry.py` for him** in Phase 3. That file is the single
  most valuable thing he writes. Checkpoint it: Resource → TracerProvider →
  BatchSpanProcessor → exporter → propagator, one at a time.
- **Creating or editing dashboards through the Grafana UI or HTTP API** as a
  deliverable. Everything must come from provisioning files or the Foundation
  SDK generator, because "destroy and recreate in seconds" is an explicit
  requirement from Babak. Read-only API calls for *verification* are fine.
- **Writing OpenTelemetry Collector YAML.** The platform uses **Alloy**, whose
  config is component-based (`otelcol.receiver.otlp`, `otelcol.exporter.otlp`,
  `loki.source.docker`, `prometheus.remote_write`), not YAML receiver/processor/
  exporter pipelines. A correct-looking Collector config is a wrong answer here.
- **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator
  already does this.
- **Assuming OTLP metrics or logs arrive.** Alloy 404s both over OTLP/HTTP, and
  the only trace of that is an export error in the app's logs. Verify in
  Prometheus / Loki before claiming anything works.
- **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned
  Loki derived field with no visible error.
- **`avg(p99)` or averaging any quantile** across instances or time buckets.
  Aggregate the histogram buckets, then compute the quantile.
- **High-cardinality metric labels** — `account_id`, `transaction_id`,
  `merchant_name`, raw paths. These belong on spans. (There is one *deliberate*
  cardinality-explosion exercise in Phase 5; that one is intentional and gets
  reverted.)
- **`constant-vus` for load profiles** where an open model is correct.
  Closed-model executors hide latency through coordinated omission.
- **Putting `trace_id` in a Loki label** instead of the log body + derived field.
- **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a
  Phase 10 judgment exercise and a small side pipeline only — not the
  architecture.
- Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md`
  for him. Those are his reasoning. Review, challenge, poke holes — don't write.

---

## Conventions

- Python 3.12, FastAPI, SQLAlchemy 2.0 async + asyncpg, `httpx` async,
  `redis-py` asyncio, Pydantic v2. No Celery, no RabbitMQ — Redis Streams,
  deliberately, because it has no automatic context propagation.
- One Docker image for all services; `CMD` differs per service.
- Semantic conventions for standard attributes (`http.request.method`,
  `db.system`, `server.address`). Namespace custom ones as `ledgerline.*`.
- `service.name` matches the directory name under `services/`.
- Duplication across services is fine. Don't refactor toward a shared framework.
- Docker Compose only, locally, through Phase 10. Kubernetes is a later
  follow-up — porting instrumentation and debugging a cluster at the same time
  is how this stalls.
- Never save a Grafana dashboard from the UI.