LedgerLine/docs/roadmap/phase-9.md

Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry

Part of docs/roadmap/ — see intro.md for platform/background context and summary.md for how every phase fits together. Prev: phase-8.md · Next: phase-10.md

Status: Not started

Message: Not technically blocked by earlier phases (it stands on its own as a judgment exercise), but sequenced last-but-one so Phase 8's incident to re-diagnose in Kibana already exists.

Time-box: 1–2 evenings.

Babak's remaining questions are all judgment questions. He isn't testing whether you can recite what Filebeat does; he's testing whether you have a framework for choosing. Here's the framework, then the answers as worked examples.

The framework — six axes to reason on

Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:

1. Where does the data originate? Inside your application (you can instrument it) or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
2. What shape is it already in? Structured JSON at the source, or unstructured text needing grok/regex parsing? Parsing cost has to land somewhere.
3. Which signals? Logs only, or logs + metrics + traces that need to correlate?
4. What's the destination, and how many? One backend forever, or several, or one you expect to change?
5. Who pays the CPU, and where's the failure boundary? Parsing at the edge (agent on every host) distributes cost and adds resilience but multiplies config surface. Parsing centrally concentrates cost and creates a single point of failure — but one place to change rules.
6. What already exists, and what does it cost? Existing Elastic investment, existing licences, team skills, and licensing constraints (Elastic's Filebeat and Logstash are source-available under Elastic licensing; the OTel Collector, Fluent Bit and Vector are permissively licensed). Migration is never free.

The answers

When to use Filebeat. When you need a lightweight shipper that tails log files from a host or container and forwards them with minimal transformation — and your destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once delivery with a registry of file offsets, backpressure handling, and prebuilt modules for common services (nginx, MySQL, system logs) that come with parsing and dashboards included. Reach for it when the source is a file you don't control the format of, the app can't be modified, and Elasticsearch is where it's going.

When to use Logstash. When you need real transformation: parsing unstructured text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB), splitting or merging events, conditional routing to multiple destinations, or buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine for events. It's heavyweight — a JVM — so you run a few of them centrally, not one per host.

When to use Filebeat and Logstash together, and why. This is the classic Elastic architecture and the reason is a division of labour between edge and centre:

• Filebeat runs on every host: cheap, tails files, handles offsets and backpressure, ships onward. You want the thing running in a thousand places to be small and boring.
• Logstash runs on a few central nodes: does the expensive grok parsing, enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so you want them in a handful of places you can redeploy, not on a thousand hosts.
• It also gives you a buffer: if Elasticsearch is down or slow, Logstash's persistent queue absorbs it and Filebeat backs off gracefully, rather than every host queuing independently or dropping.

So: Filebeat for collection and delivery, Logstash for processing and routing. Use both when you have many hosts and non-trivial parsing/enrichment and need central buffering. Skip Logstash when your logs are already structured JSON and Elasticsearch ingest pipelines can do the little parsing that remains.

When to use OpenTelemetry.

• You own the application code and want telemetry generated from inside it — spans with business context, custom metrics, logs already carrying trace IDs.
• You need more than one signal, correlated. This is the strongest argument: one SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs threaded through all three. Neither Filebeat nor Logstash does traces.
• You have a polyglot estate — Python, Go, TypeScript — and want one consistent data model across all of them.
• You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for Jaeger by editing Collector YAML.
• You need distributed tracing at all. There is no serious vendor-neutral alternative.
• Your telemetry needs processing in flight — redaction, tail sampling, attribute rewriting — without touching the app.

When not to use OpenTelemetry. This is the question that separates people who have used it from people who have read about it.

• You don't own the code. A network appliance, a SaaS webhook, a vendor binary, an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a file-based collector.
• You only need one signal from one source with no correlation. A single nginx access log going to a single log store: OTel is more machinery than the job needs. Fluent Bit or Filebeat is simpler and finishes the job today.
• You're deep in a committed ecosystem and the migration doesn't pay. A mature Elastic estate with hundreds of Beats modules, dashboards and detection rules is years of investment. "We should use OTel because it's the standard" is not a business case. Note the direction of travel though: the Collector can already receive from and export to most of these, so a gradual migration is usually the right shape — often by putting a Collector or Vector in front of the existing agents rather than ripping them out.
• Hard resource constraints. Embedded, edge, or extremely tight sidecar budgets, where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
• The signal is genuinely business analytics, not telemetry. Clickstream data for a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different retention, different query patterns, different cost model.
• Maturity gaps for a specific need in a specific language. OTel logs support has matured at different rates per language, and profiling is still alpha. Check the status page for your language before betting on a signal.
• Team capacity. OTel has real conceptual surface area — you've just spent weeks on it. A two-person team that needs error tracking by Friday should install Sentry.

The senior framing to give Babak: these tools aren't competitors so much as different layers. OpenTelemetry is a generation and transport standard for telemetry you own. Filebeat and Fluent Bit are collectors for telemetry you don't. Logstash and Vector are processing engines. Loki, Tempo, Prometheus and Elasticsearch are stores. Most real architectures use one from each layer, and the interesting question is always which layer a given problem lives in.

Resources

• Better Stack — Filebeat vs Logstash: How to Choose (free article).
• SigNoz — OpenTelemetry vs ELK Stack (free article).
• OpenTelemetry docs → Collector → filelog receiver and transform processor — read these to see how the Collector does the Filebeat/Logstash job itself.
• OpenTelemetry docs → Instrumentation status by language — for the maturity caveats.

Build — and this is no longer hypothetical

Your platform repo ships a complete second stack: playbooks/elk-grafana.yml deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with Filebeat reading Docker container logs and Logstash parsing JSON into daily logs-YYYY.MM.dd indices. That is almost certainly why Babak asked these questions — the codebase he handed you contains both worlds.

1. ansible-playbook playbooks/elk-grafana.yml. Now the same Ledgerline logs flow down two pipelines: Filebeat → Logstash → Elasticsearch, and Alloy → Loki. Compare them on the same data.
2. Test the direct ingest path the README documents — pipe a JSON line into Logstash's TCP input on 5000, then query Elasticsearch — so you understand where parsing happens before you change it.
3. Keep a copy of the Phase 0 plain-text logs and write a Logstash grok filter that parses them. Then do the same job with Alloy's loki.process stages. Time both. Note which one you'd rather debug at 2 a.m.
4. Compare the investigation experience directly: take one incident you already diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL for some questions, and Loki's cost model genuinely beats Elasticsearch's for others.
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are permissively licensed. For some employers that alone decides it.
6. Write docs/PIPELINE-CHOICES.md: the six axes, the answers above, and — most valuably — what you actually found annoying about each. Having run both stacks against the same application is the difference between reciting a comparison and arguing one.