Movses Aslanyan
eb99cdbb8c
Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing Ledgerline as an observability learning project built on a deliberately imperfect seven-service payments backend.
4.7 KiB
4.7 KiB
Roadmap summary — all phases at a glance
Part of docs/roadmap/ — the one-page overview. Read intro.md
first for platform/background context; read the individual phase-N.md file
for the phase you're actually working on. This file exists so an agent (or
you) can check overall status and sequencing without loading every phase's
full detail.
Eleven phases (0 through 10 — Phase 0 has two halves, 0a and 0b). Each phase file follows the same shape: Status, Message (what's done, notes, warnings), Time-box, Concepts, Resources, Build, Done when.
Status and focus
| Phase | File | Status | Focus | Time-box | Main output |
|---|---|---|---|---|---|
| 0a | phase-0.md |
Done | Integration test suite (agent, tests only) | 1 evening | Red suite + RED-BASELINE.md |
| 0b | phase-0.md |
Done | Base project implemented to green (second agent) | 1–2 evenings | Ledgerline running, list of frustrations |
| 1 | phase-1.md |
Not started | Concepts + first trace | 1 evening | One trace in Tempo |
| 2 | phase-2.md |
Not started | Distributed tracing | 2–3 evenings | Connected trace across 7 services incl. the async hop |
| 3 | phase-3.md |
Not started | Structured logs | 1–2 evenings | Two-way trace↔log correlation |
| 4 | phase-4.md |
Not started | Metrics | 2 evenings | RED per service, service map, exemplars |
| 5 | phase-5.md |
Not started | Dashboards as code | 2 evenings | make obs-reset |
| 6 | phase-6.md |
Not started | k6 + percentiles | 1–2 evenings | Baseline vs degraded load profiles |
| 7 | phase-7.md |
Not started | SLO / error budget / RTO / RPO | 1–2 evenings | docs/SLO.md + burn-rate alerts |
| 8 | phase-8.md |
Not started | Bottleneck hunting | 2 evenings | docs/INVESTIGATION.md |
| 9 | phase-9.md |
Not started | Filebeat / Logstash / OTel judgment | 1–2 evenings | docs/PIPELINE-CHOICES.md |
| 10 | phase-10.md |
Not started | Delivery | 1 evening | README + rehearsed demo |
What each phase is, in one line
- 0 — Build the base project, test-first. Write the black-box test suite,
then implement the seven-service system against it as a frozen contract.
Done — see
phase-0.mdfor the full implementation report. - 1 — The model, and one signal end to end. Learn the OTel API/SDK/Collector split and the M.E.L.T. vocabulary; deploy the real Grafana-stack platform via Ansible and get one auto-instrumented trace into Tempo.
- 2 — Distributed tracing. The heart of the task: auto- and manual-instrument all seven services, and hand-propagate trace context across the Redis Stream hop, which carries none by default.
- 3 — Structured logs, correlated to traces. Replace prose logging with
structured events carrying a lowercase
trace_id, so Loki and Tempo link in both directions. - 4 — Metrics. Instrument RED metrics per service, understand cardinality limits, and route app metrics to Prometheus given Alloy drops OTLP metrics silently. Tempo's metrics-generator already produces span metrics and the service graph — don't duplicate it in the collector.
- 5 — Dashboards and alerts as code. Generate dashboard JSON with the Grafana Foundation SDK and ship it through the Ansible role, so "destroy and recreate in seconds" is a real, repeatable command.
- 6 — Load testing with k6, and percentiles. Learn why percentiles can't be averaged, what coordinated omission is, and drive Ledgerline with arrival-rate executors to get a clean before/after load profile.
- 7 — SLI, SLO, SLA, error budgets, RTO, RPO. Define Ledgerline's SLIs and SLOs from the Phase 6 baseline (not from thin air), write an error-budget policy, and wire multi-burn-rate alerts.
- 8 — Bottleneck hunting (the capstone). Given a random, undisclosed set of chaos flags, find the pathologies from telemetry alone using a top-down, hypothesis-driven method, and write it up as an incident doc.
- 9 — Tool-selection judgment. Reason through Filebeat vs Logstash vs OpenTelemetry on six concrete axes, using the platform's real second stack (Filebeat → Logstash → Elasticsearch → Kibana) as the counterpoint.
- 10 — Package the delivery. README, a rehearsed five-minute demo, the
written deliverables (
SLO.md,PIPELINE-CHOICES.md,INVESTIGATION.md,CONCEPTS.md), and an honest "what I'd do differently in production" section.
If you have to cut scope
Cut Phase 9's build (keep the writing) and Phase 6's soak test first. Do not cut Phase 8 — it's the only phase that proves you learned anything.