Add project documentation and Claude instructions

Roadmap, phase specs, pathology catalog, and CLAUDE.md establishing
Ledgerline as an observability learning project built on a deliberately
imperfect seven-service payments backend.

docs/roadmap/phase-9.md

@@ -0,0 +1,161 @@
+# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
+
+*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
+
+**Status:** Not started
+
+**Message:** Not technically blocked by earlier phases (it stands on its own
+as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
+re-diagnose in Kibana already exists.
+
+**Time-box:** 1–2 evenings.
+
+Babak's remaining questions are all *judgment* questions. He isn't testing whether
+you can recite what Filebeat does; he's testing whether you have a framework for
+choosing. Here's the framework, then the answers as worked examples.
+
+### The framework — six axes to reason on
+
+Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
+
+1. **Where does the data originate?** Inside your application (you can instrument it)
+   or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
+   touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
+2. **What shape is it already in?** Structured JSON at the source, or unstructured
+   text needing grok/regex parsing? Parsing cost has to land *somewhere*.
+3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
+4. **What's the destination, and how many?** One backend forever, or several, or one
+   you expect to change?
+5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
+   on every host) distributes cost and adds resilience but multiplies config surface.
+   Parsing centrally concentrates cost and creates a single point of failure — but
+   one place to change rules.
+6. **What already exists, and what does it cost?** Existing Elastic investment,
+   existing licences, team skills, and licensing constraints (Elastic's Filebeat and
+   Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
+   Bit and Vector are permissively licensed). Migration is never free.
+
+### The answers
+
+**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
+from a host or container and forwards them with minimal transformation — and your
+destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
+delivery with a registry of file offsets, backpressure handling, and prebuilt modules
+for common services (nginx, MySQL, system logs) that come with parsing and dashboards
+included. Reach for it when the source is a file you don't control the format of, the
+app can't be modified, and Elasticsearch is where it's going.
+
+**When to use Logstash.** When you need real *transformation*: parsing unstructured
+text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
+splitting or merging events, conditional routing to multiple destinations, or
+buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
+for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
+per host.
+
+**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
+architecture and the reason is a **division of labour between edge and centre**:
+
+- Filebeat runs on *every* host: cheap, tails files, handles offsets and
+  backpressure, ships onward. You want the thing running in a thousand places to be
+  small and boring.
+- Logstash runs on a *few* central nodes: does the expensive grok parsing,
+  enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
+  you want them in a handful of places you can redeploy, not on a thousand hosts.
+- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
+  persistent queue absorbs it and Filebeat backs off gracefully, rather than every
+  host queuing independently or dropping.
+
+So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
+Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
+need central buffering. Skip Logstash when your logs are already structured JSON and
+Elasticsearch ingest pipelines can do the little parsing that remains.
+
+**When to use OpenTelemetry.**
+
+- You own the application code and want telemetry generated *from inside it* — spans
+  with business context, custom metrics, logs already carrying trace IDs.
+- You need **more than one signal**, correlated. This is the strongest argument: one
+  SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
+  threaded through all three. Neither Filebeat nor Logstash does traces.
+- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
+  data model across all of them.
+- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
+  Jaeger by editing Collector YAML.
+- You need distributed tracing at all. There is no serious vendor-neutral alternative.
+- Your telemetry needs processing in flight — redaction, tail sampling, attribute
+  rewriting — without touching the app.
+
+**When *not* to use OpenTelemetry.** This is the question that separates people who
+have used it from people who have read about it.
+
+- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
+  an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
+  file-based collector.
+- **You only need one signal from one source with no correlation.** A single nginx
+  access log going to a single log store: OTel is more machinery than the job needs.
+  Fluent Bit or Filebeat is simpler and finishes the job today.
+- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
+  Elastic estate with hundreds of Beats modules, dashboards and detection rules is
+  years of investment. "We should use OTel because it's the standard" is not a
+  business case. Note the direction of travel though: the Collector can already
+  receive from and export to most of these, so a *gradual* migration is usually the
+  right shape — often by putting a Collector or Vector in front of the existing
+  agents rather than ripping them out.
+- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
+  where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
+- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
+  a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
+  retention, different query patterns, different cost model.
+- **Maturity gaps for a specific need in a specific language.** OTel logs support has
+  matured at different rates per language, and profiling is still alpha. Check the
+  status page for your language before betting on a signal.
+- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
+  on it. A two-person team that needs error tracking by Friday should install Sentry.
+
+The senior framing to give Babak: *these tools aren't competitors so much as
+different layers.* OpenTelemetry is a **generation and transport standard** for
+telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
+don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
+Elasticsearch are **stores**. Most real architectures use one from each layer, and
+the interesting question is always which layer a given problem lives in.
+
+### Resources
+
+- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
+- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
+- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
+  read these to see how the Collector does the Filebeat/Logstash job itself.
+- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
+  caveats.
+
+### Build — and this is no longer hypothetical
+
+Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
+deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
+Filebeat reading Docker container logs and Logstash parsing JSON into daily
+`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
+questions — the codebase he handed you contains both worlds.
+
+1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
+   flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
+   Alloy → Loki. Compare them on the same data.
+2. Test the direct ingest path the README documents — pipe a JSON line into
+   Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
+   parsing happens before you change it.
+3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
+   that parses them. Then do the same job with Alloy's `loki.process` stages. Time
+   both. Note which one you'd rather debug at 2 a.m.
+4. Compare the *investigation experience* directly: take one incident you already
+   diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
+   you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
+   wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
+   for some questions, and Loki's cost model genuinely beats Elasticsearch's for
+   others.
+5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
+   licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
+   permissively licensed. For some employers that alone decides it.
+6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
+   valuably — what you actually found annoying about each. Having run both stacks
+   against the same application is the difference between reciting a comparison and
+   arguing one.
+