LedgerLine/docs/roadmap/phase-10.md

# Phase 10 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry

*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-9.md`](phase-9.md) · Next: [`phase-11.md`](phase-11.md)*

**Status:** Not started

**Message:** Not technically blocked by earlier phases (it stands on its own
as a judgment exercise), but sequenced last-but-one so Phase 9's incident to
re-diagnose in Kibana already exists.

**Time-box:** 1–2 evenings.

Babak's remaining questions are all *judgment* questions. He isn't testing whether
you can recite what Filebeat does; he's testing whether you have a framework for
choosing. Here's the framework, then the answers as worked examples.

### The framework — six axes to reason on

Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:

1. **Where does the data originate?** Inside your application (you can instrument it)
   or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
   touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
2. **What shape is it already in?** Structured JSON at the source, or unstructured
   text needing grok/regex parsing? Parsing cost has to land *somewhere*.
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
4. **What's the destination, and how many?** One backend forever, or several, or one
   you expect to change?
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
   on every host) distributes cost and adds resilience but multiplies config surface.
   Parsing centrally concentrates cost and creates a single point of failure — but
   one place to change rules.
6. **What already exists, and what does it cost?** Existing Elastic investment,
   existing licences, team skills, and licensing constraints (Elastic's Filebeat and
   Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
   Bit and Vector are permissively licensed). Migration is never free.

### The answers

**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
from a host or container and forwards them with minimal transformation — and your
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
included. Reach for it when the source is a file you don't control the format of, the
app can't be modified, and Elasticsearch is where it's going.

**When to use Logstash.** When you need real *transformation*: parsing unstructured
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
splitting or merging events, conditional routing to multiple destinations, or
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
per host.

**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
architecture and the reason is a **division of labour between edge and centre**:

- Filebeat runs on *every* host: cheap, tails files, handles offsets and
  backpressure, ships onward. You want the thing running in a thousand places to be
  small and boring.
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
  enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
  you want them in a handful of places you can redeploy, not on a thousand hosts.
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
  persistent queue absorbs it and Filebeat backs off gracefully, rather than every
  host queuing independently or dropping.

So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
need central buffering. Skip Logstash when your logs are already structured JSON and
Elasticsearch ingest pipelines can do the little parsing that remains.

**When to use OpenTelemetry.**

- You own the application code and want telemetry generated *from inside it* — spans
  with business context, custom metrics, logs already carrying trace IDs.
- You need **more than one signal**, correlated. This is the strongest argument: one
  SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
  threaded through all three. Neither Filebeat nor Logstash does traces.
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
  data model across all of them.
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
  Jaeger by editing Collector YAML.
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
  rewriting — without touching the app.

**When *not* to use OpenTelemetry.** This is the question that separates people who
have used it from people who have read about it.

- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
  an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
  file-based collector.
- **You only need one signal from one source with no correlation.** A single nginx
  access log going to a single log store: OTel is more machinery than the job needs.
  Fluent Bit or Filebeat is simpler and finishes the job today.
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
  Elastic estate with hundreds of Beats modules, dashboards and detection rules is
  years of investment. "We should use OTel because it's the standard" is not a
  business case. Note the direction of travel though: the Collector can already
  receive from and export to most of these, so a *gradual* migration is usually the
  right shape — often by putting a Collector or Vector in front of the existing
  agents rather than ripping them out.
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
  where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
  a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
  retention, different query patterns, different cost model.
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
  matured at different rates per language, and profiling is still alpha. Check the
  status page for your language before betting on a signal.
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
  on it. A two-person team that needs error tracking by Friday should install Sentry.

The senior framing to give Babak: *these tools aren't competitors so much as
different layers.* OpenTelemetry is a **generation and transport standard** for
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
Elasticsearch are **stores**. Most real architectures use one from each layer, and
the interesting question is always which layer a given problem lives in.

### Resources

- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
  read these to see how the Collector does the Filebeat/Logstash job itself.
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
  caveats.

### Build — and this is no longer hypothetical

Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
Filebeat reading Docker container logs and Logstash parsing JSON into daily
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
questions — the codebase he handed you contains both worlds.

1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
   flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
   Alloy → Loki. Compare them on the same data.
2. Test the direct ingest path the README documents — pipe a JSON line into
   Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
   parsing happens before you change it.
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
   that parses them. Then do the same job with Alloy's `loki.process` stages. Time
   both. Note which one you'd rather debug at 2 a.m.
4. Compare the *investigation experience* directly: take one incident you already
   diagnosed in Phase 9 and re-run it in Kibana Discover. What can you do there that
   you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
   wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
   for some questions, and Loki's cost model genuinely beats Elasticsearch's for
   others.
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
   licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
   permissively licensed. For some employers that alone decides it.
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
   valuably — what you actually found annoying about each. Having run both stacks
   against the same application is the difference between reciting a comparison and
   arguing one.