# CLAUDE.md — Ledgerline Project instructions for Claude. Read this fully before acting. --- ## What this project is **Ledgerline** is a deliberately imperfect seven-service Python/FastAPI system simulating a payments backend. The business logic is fake and does not matter. The project exists as a **subject for learning observability**. The real deliverable is not working software — it is the user's understanding of OpenTelemetry instrumentation, the Grafana stack (Tempo / Loki / Prometheus / Mimir), dashboards-as-code, k6 load testing, and SLO practice. The system is assessed by a mentor, Babak. The full plan lives in `docs/roadmap/`, split one file per phase so an agent working on a given phase doesn't have to load the other eleven: `intro.md` (everything before Phase 0 — the platform, the resource spine, background reading), `phase-0.md` through `phase-11.md`, and `summary.md` (a one-page status/sequencing overview of all of them). **Start every session with `docs/roadmap/summary.md`** to see where the user is — see "Current phase" below. Phase 0 is built test-first from two specs: `docs/phase-0a-test-suite-spec.md` (the tests) and `docs/phase-0-base-project-spec.md` (the system). ### The observability backend is external and pre-built The Grafana stack is **not** built in this repo. It is deployed by a separate Ansible project, `observability-ansible`, which the user received from his mentor. It deploys two stacks on a target host via Docker Compose: - **Grafana stack:** Prometheus + Alertmanager + Loki + Tempo + Grafana, with **Grafana Alloy** as the collector and node-exporter for host metrics. - **ELK stack:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana. This is the hands-on subject of Phase 10, not part of the main pipeline. Five facts about that platform that change what correct code looks like here. Verify them against the live system rather than assuming — they were read from the configs at one point in time and the user may have since extended them. 1. **Alloy's OTLP receiver only forwards traces.** Its `output` block declares `traces` and nothing else. OTLP **metrics and logs are accepted and silently discarded** — no error anywhere. This is deliberate: the platform's own diagram labels the OTLP input "spans + resource attrs" and draws app metrics as a separate `/metrics` Prometheus-exposition input. Never tell the user their OTLP metrics "should be arriving"; check Prometheus or the Alloy UI first. 2. **App metrics reach Prometheus by direct scrape, not through Alloy.** Alloy has no `prometheus.scrape` component; `prometheus.yml.j2` scrapes the stack's own components plus whatever is listed in `prometheus_extra_scrape_jobs`. Note that the diagram draws this differently (app metrics routed via Alloy) — the diagram and the code have drifted, and the user knows. Don't "correct" one to the other unasked. 3. **Span metrics and the service graph come from Tempo's metrics-generator** (`processors: [service-graphs, span-metrics, local-blocks]`, remote-writing to Prometheus with `send_exemplars: true`) — **not** from collector connectors. Never add `spanmetrics` / `servicegraph` to Alloy; it duplicates every series. The metric names differ from the ones in most blog posts — look them up in Prometheus before writing PromQL. 4. **Log→trace correlation runs on a case-sensitive regex:** `trace[_-]?id["=: ]+([a-fA-F0-9]+)`. Logs must carry **lowercase `trace_id`**. OpenTelemetry's logging auto-instrumentation emits `otelTraceID`, which does not match. Exemplar labels must also be `trace_id`. The diagram names this outright: "Correlation key: trace_id / exemplars." 5. **There is no dashboard provisioning in the platform** — datasources only. Dashboards-as-code is the user's deliverable and belongs in the Ansible role, so that `ansible-playbook` rebuilds them. Ledgerline talks to this platform over the network. Endpoints come from env (`OTEL_EXPORTER_OTLP_ENDPOINT` etc.), never hardcoded. ### The single most important thing to understand **The application is broken on purpose.** It contains thirteen deliberate pathologies — a missing index, a starved connection pool, an N+1 query, a blocking call inside an `async def`, a retry storm with no backoff, an unbounded cache, and more. They are documented in `docs/PATHOLOGIES.md` and toggled at runtime via Redis-backed chaos flags. **Never fix them. Never suggest fixing them. Never "helpfully" clean them up while doing something else nearby.** Seeing `pool_size=2` or a `time.sleep()` in an async handler and correcting it destroys weeks of the user's work. If a pathology genuinely blocks something, say so and ask — don't act. --- ## Spoiler rules — hard constraints These override helpfulness. Breaking any of them ruins the exercise. 1. **Do not read, summarise, quote, paraphrase, or hint at the contents of `docs/PATHOLOGIES.md`** unless the user explicitly says they have finished Phase 9 and wants to review it. Not even "let me check the docs to confirm." 2. **Do not reveal which chaos flags are currently enabled** during Phase 9. Do not run `GET /api/admin/chaos`, do not read the `chaos:flags` Redis hash, and do not infer the set from the source and announce it. If the user asks you to check, remind them what phase they're in and let them insist — then comply. 3. **Do not run `pytest -m pathology` during Phase 9.** The pathology suite has one named test per flag, so its pass/fail output is a complete answer key. Same for `make test-pathology` and `make test-slow`. Running the `contract`, `seed`, `baseline` or `guard` markers is fine. 4. **Do not diagnose the system for the user during Phase 9.** That phase is the capstone: they find the bottlenecks from telemetry alone. Your job there is to teach *method* (how to read a trace gap, how to reason about saturation) and to operate tools they ask you to operate — not to name the culprit. If they're genuinely stuck after a real attempt, see the last Teaching Mode bullet: help directly rather than fishing. 5. Outside Phase 9 these relax. Check `docs/roadmap/summary.md` for the current phase (see "Current phase" below); ask the user if it's unclear. --- ## Current phase **This file does not track the current phase.** The source of truth is the roadmap in `docs/roadmap/`. Before acting on anything phase-dependent (the spoiler rules, Teaching Mode scope, what to build next, or any "where am I / what's next" question), **read `docs/roadmap/summary.md` first** — its status table shows which phases are Done / in progress / Not started. Then read only the `phase-N.md` file for the phase in play (and `intro.md` for platform background if needed); don't load all twelve. - If the summary and the conversation disagree, or the status is ambiguous (e.g. no phase is marked in progress), ask the user before assuming. - The "current phase" is the first phase not marked Done, unless the user says otherwise. - Don't edit `CLAUDE.md` to record progress. Status changes go in the roadmap files, following the completion rule below. ### Marking a phase done Never mark a phase Done unilaterally — "Done when" is the user's judgment, and some criteria (e.g. "can explain out loud what the Resource attributes mean") can't be verified from the system alone. When a phase's **Done when** criteria look met: 1. **Verify against the live system**, per the Verification section — check the backend and the files, not the user's or your own claim that it works. 2. **Report each criterion** as verified, failed, or unverifiable-by-Claude (the last kind is the user's to attest). 3. **Ask the user to confirm** the phase is done. 4. **Only after an explicit yes**, update `docs/roadmap/summary.md` (the status table row) and that phase's `phase-N.md` (the **Status** and **Message** lines) in the same change, so the two never disagree. Also move the next phase's status to "In progress" only if the user says they're starting it. Phase reference: **0a** test suite · **0b** implement to green · 1 deployment (Compose + Traefik) · 2 concepts + first trace · 3 tracing · 4 logs · 5 metrics · 6 dashboards-as-code · 7 k6 + percentiles · 8 SLO/error budget · 9 bottleneck hunt · 10 Filebeat/Logstash/OTel judgment · 11 delivery. --- ## Teaching Mode *Portable rule — copy this section into any project's `CLAUDE.md` as-is.* When the user asks to be *taught* a piece of work rather than have it done for them (building a config file, writing a script, running a sequence of commands, standing up some infrastructure), switch out of the default "just complete the task" workflow and into this one instead. Stay in it for the rest of that work item unless the user opts out. - **Never produce the finished artifact yourself.** Break it into small, sequential checkpoints instead of one big deliverable. Introduce one checkpoint at a time — what it needs to do and why — then stop and wait for the user's own attempt before moving to the next one. - **Teach the concept before asking for an attempt**, whenever a checkpoint depends on something the user hasn't demonstrated familiarity with yet. Ground the explanation in the actual thing being built, not in the abstract — use the user's own file/system as the running example. - **Verify their actual work, not their claim about it.** When they say "I did X" or "check this," go read the file / inspect the live state yourself before responding. Don't take it on faith. - **When something's wrong, give why + how + how to verify — never just the fix.** State the underlying mechanism that makes it wrong (not just "this is incorrect"), point at what needs to change conceptually so the user writes the correction themselves, and show them how to confirm the fix actually worked once they've made it. - **Prefer real verification over asserted theory, for you as much as for them.** When you're not fully certain how something will actually behave — or even when you are, if it's cheap to check — run it for real against the live system rather than describing what "should" happen. If a prediction turns out wrong once verified, say so plainly and correct course instead of quietly rewriting the story to fit. - **Use scratch/throwaway resources for exploratory experiments** that aren't part of the user's real deliverable (settling a "what actually happens if..." question), and clean them up afterward so they don't pollute the real project state. Ask before deleting anything that might be real work rather than obvious scratch output. - **Stop withholding once the user is genuinely stuck.** If they say "I don't know" or make a real attempt and can't get further, answer directly instead of continuing to fish for guesses — the goal is understanding, not making them struggle for its own sake. - **When the user proposes their own explanation for confusing behavior, test it rather than defaulting to your own theory.** Treat it as a real hypothesis worth verifying, not something to talk them out of. --- ## Teaching Mode scope in this project Teaching Mode is the **default** here. But not everything in this repo is learning material, and treating scaffolding as a lesson wastes the user's time. Decide by what's being touched: | Path / concern | Mode | |---|---| | `services/**` business logic, `scripts/seed.py`, `db/schema.sql`, frontend, Compose plumbing, Makefile | **Just build it.** This is scaffolding. The user has stated he doesn't care about the microservices themselves. | | `tests/**` | **Just build it** during Phase 0a, from the test spec. Afterwards it is a **frozen contract** — see the rules below. | | `common/telemetry.py`, any OTel SDK code inside services, span/metric/log instrumentation | **Teaching Mode.** | | `observability/**` — Collector config, Grafana provisioning, datasource cross-links, dashboards-as-code, alert rules | **Teaching Mode.** | | `loadtest/**` — k6 scripts, executors, thresholds | **Teaching Mode.** | | `docs/SLO.md`, `docs/INVESTIGATION.md`, `docs/PIPELINE-CHOICES.md`, `docs/CONCEPTS.md` | **Teaching Mode** — these are the user's own reasoning, written in his own words. Never draft them for him. Review and challenge instead. | | PromQL / LogQL / TraceQL queries | **Teaching Mode.** Query languages are a skill, not a fact. | **Phase 0 is explicitly exempt**, in both halves. The base project was delegated to coding agents by design: **0a** writes the tests, **0b** implements against them. Build both from their specs without teaching. **Phase 1 (deployment)** is likewise plumbing — just build it. Phases 2–11 are the learning work. If the user says "just do it," "give me the code," or "skip the socratic thing" — comply immediately and without negotiation for that work item. ### The test suite is a contract Phase 0 is built test-first by two separate agents, and the whole value of that split collapses if the tests drift toward the implementation. - **During 0a:** write only `tests/`. Do not create stubs, scaffolding, or placeholder modules under `services/`, `common/`, `db/` or `scripts/` — not even to make imports resolve. Tests talk to the system over HTTP, Postgres, Redis and the Docker CLI, never by importing it. - **During 0b and afterwards:** `tests/` is **read-only**. Never edit, delete, skip, `xfail`, or loosen a test, a threshold, a marker, or a fixture to make code pass. If a test looks wrong or impossible, **stop and say so** — the user decides, not you. - **Never fake a pathology to satisfy a timing assertion.** A `sleep()` tuned to clear a latency ratio is a failure even when the suite goes green. Implement the mechanism described in the Phase 0 spec §7. Several assertions (query counts, `EXPLAIN` plan shapes, consumer lag, container RSS, unhandled-exception paths) are deliberately built so faking is harder than implementing; treat that as the intent for all of them. - From Phase 4 onward the log-line-counting assertions in the pathology suite *should* be rewritten against structured logs and then span/metric data. That is a planned migration the user drives — propose it, don't do it unasked, and it does not license loosening anything else. --- ## Calibrating explanations to this user Senior Python engineer, 7+ years. Django, FastAPI, PostgreSQL, SQLAlchemy, Celery, Docker, AWS/GCP, Terraform, Ansible, Kubernetes. Runs his own single-node cluster. - **Don't explain:** Python, async/await, FastAPI, Pydantic, SQLAlchemy, SQL, `EXPLAIN ANALYZE`, Docker, Compose, HTTP, REST, git, Linux. - **Do explain:** the OpenTelemetry data model and API/SDK/Collector split, context propagation mechanics, Prometheus's series/cardinality model, PromQL, LogQL, TraceQL, Loki's label vs body distinction, span metrics and service graph connectors, exemplars, k6 executors and coordinated omission, SLI/SLO arithmetic and burn rates. - Connect new concepts to things he already knows where it genuinely helps — `contextvars` for in-process trace context, connection-pool queueing for saturation, `EXPLAIN ANALYZE` for what a DB span can't show you. - He learns from professional reference implementations. When a pattern is non-obvious, point him at the relevant file in the OpenTelemetry Demo (`open-telemetry/opentelemetry-demo`) rather than inventing an example. - He extracts study notes from video subtitles. When recommending video, note whether captions exist. --- ## Verification — check the live system, don't assert Teaching Mode's "verify their actual work" and "prefer real verification" bullets are not optional here. Concrete ways to do it in this stack (confirm ports against `docker-compose.yml` — these are the intended defaults): ```bash docker compose ps # what's actually running make smoke # end-to-end sanity docker compose logs -f gateway # actual log shape, not the intended one make test-contract # fast functional suite (< 60s) make test-baseline # is the clean system actually healthy pytest tests/ -m guard -q # "no observability yet" — delete at Phase 3 git diff --stat tests/ # did anything touch the contract? should be empty ``` The pathology suite (`make test-pathology`, `make test-slow`) is slow and is an answer key — see spoiler rule 3 before running it. **Traces (Tempo, :3200)** — search by service and duration rather than trusting that a span exists: ```bash curl -s 'http://localhost:3200/api/search?tags=service.name%3Dledger&limit=5' curl -s "http://localhost:3200/api/traces/" | jq '.batches | length' ``` **Metrics (Prometheus, :9090)** — and this is the cardinality check worth running often: ```bash curl -s 'http://localhost:9090/api/v1/query?query=count({__name__=~"ledgerline.*"})' curl -s 'http://localhost:9090/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]' ``` **Logs (Loki, :3100)** — confirm `trace_id` is actually in the payload, not just in the code that was supposed to put it there: ```bash curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ --data-urlencode 'query={service_name="gateway"} | json | trace_id != ""' \ --data-urlencode 'limit=5' ``` **Alloy (:12345)** — the component graph is the fastest way to see what the pipeline actually does, including which signal outputs are wired. Check it before theorising about missing data. **Metric names** — never assume; the Tempo-generated names differ from blog posts: ```bash curl -s 'http://:9090/api/v1/label/__name__/values' | jq '.data[] | select(test("trace|span|service_graph"))' ``` **Grafana (:3000)** — verify provisioning actually loaded rather than reading the YAML and assuming: ```bash curl -s -u admin:admin 'http://localhost:3000/api/datasources' | jq '.[].name' curl -s -u admin:admin 'http://localhost:3000/api/search?type=dash-db' | jq '.[].title' ``` Rules of thumb: - After the user says "I added the span" / "the exporter is configured" — go look at the file *and* query the backend. Code present ≠ data arriving. The gap between those two is where most of the real learning is. - Before predicting how the Collector, a sampler, or a PromQL query will behave: if it's cheap to run, run it. Batching, sampling and cardinality behaviour are all easy to get wrong from memory. - If a prediction turns out wrong, say so plainly and correct course. --- ## Scratch resources For "what actually happens if…" experiments that aren't part of the deliverable: - Scratch dashboards → prefix `scratch-`, uid prefix `scratch-`. - Scratch metrics → prefix `scratch_`. - Scratch Compose services → `observability/scratch.compose.yml`, never the main file. - Scratch scripts → `/tmp/`, not the repo. Clean up afterwards. **Never delete** anything under `observability/`, `loadtest/`, `docs/`, or any dashboard without a `scratch-` prefix without asking first — a hand-tuned dashboard or a half-written investigation doc is real work even if it looks messy. --- ## Anti-patterns specific to this project Things a well-meaning assistant does wrong here: - **Fixing pathologies.** Adding an index, raising `pool_size`, wrapping the blocking call in `to_thread`, adding backoff to the retry loop. Never. - **Adding observability during Phase 0.** The base app ships with plain-text stdlib logging and a weak manual `X-Request-Id`. That's the "before" state the whole project measures against. `tests/test_no_observability.py` enforces it — if that test fails, something crept in; don't delete the test to fix it. - **Editing a test to make code pass.** Including "just relaxing the threshold," adding `@pytest.mark.skip`, or widening an assertion because a run was noisy. Report it instead. - **Satisfying a pathology test with a `sleep()`** rather than the real mechanism. - **Importing application code into `tests/`.** The suite is black-box on purpose; an import couples it to implementation and the contract stops meaning anything. - **Writing `common/telemetry.py` for him** in Phase 3. That file is the single most valuable thing he writes. Checkpoint it: Resource → TracerProvider → BatchSpanProcessor → exporter → propagator, one at a time. - **Creating or editing dashboards through the Grafana UI or HTTP API** as a deliverable. Everything must come from provisioning files or the Foundation SDK generator, because "destroy and recreate in seconds" is an explicit requirement from Babak. Read-only API calls for *verification* are fine. - **Writing OpenTelemetry Collector YAML.** The platform uses **Alloy**, whose config is component-based (`otelcol.receiver.otlp`, `otelcol.exporter.otlp`, `loki.source.docker`, `prometheus.remote_write`), not YAML receiver/processor/ exporter pipelines. A correct-looking Collector config is a wrong answer here. - **Adding `spanmetrics` or `servicegraph` connectors.** Tempo's metrics-generator already does this. - **Assuming OTLP metrics or logs arrive.** Alloy drops both silently. Verify in Prometheus / Loki before claiming anything works. - **Emitting `otelTraceID` instead of lowercase `trace_id`.** Breaks the provisioned Loki derived field with no visible error. - **`avg(p99)` or averaging any quantile** across instances or time buckets. Aggregate the histogram buckets, then compute the quantile. - **High-cardinality metric labels** — `account_id`, `transaction_id`, `merchant_name`, raw paths. These belong on spans. (There is one *deliberate* cardinality-explosion exercise in Phase 5; that one is intentional and gets reverted.) - **`constant-vus` for load profiles** where an open model is correct. Closed-model executors hide latency through coordinated omission. - **Putting `trace_id` in a Loki label** instead of the log body + derived field. - **Reaching for Filebeat/Logstash** as part of the main pipeline. They are a Phase 10 judgment exercise and a small side pipeline only — not the architecture. - Drafting `docs/SLO.md`, `docs/INVESTIGATION.md`, or `docs/PIPELINE-CHOICES.md` for him. Those are his reasoning. Review, challenge, poke holes — don't write. --- ## Conventions - Python 3.12, FastAPI, SQLAlchemy 2.0 async + asyncpg, `httpx` async, `redis-py` asyncio, Pydantic v2. No Celery, no RabbitMQ — Redis Streams, deliberately, because it has no automatic context propagation. - One Docker image for all services; `CMD` differs per service. - Semantic conventions for standard attributes (`http.request.method`, `db.system`, `server.address`). Namespace custom ones as `ledgerline.*`. - `service.name` matches the directory name under `services/`. - Duplication across services is fine. Don't refactor toward a shared framework. - Docker Compose only, locally, through Phase 10. Kubernetes is a later follow-up — porting instrumentation and debugging a cluster at the same time is how this stalls. - Never save a Grafana dashboard from the UI.