Add deployment phase and Traefik compose override
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b2112e55e1
commit
5e514ecf25
+74
-145
@@ -1,161 +1,90 @@
|
||||
# Phase 9 — Tool-selection judgment: Filebeat, Logstash, OpenTelemetry
|
||||
# Phase 9 — Bottleneck hunting (the capstone)
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-8.md`](phase-8.md) · Next: [`phase-10.md`](phase-10.md)*
|
||||
|
||||
**Status:** Not started
|
||||
|
||||
**Message:** Not technically blocked by earlier phases (it stands on its own
|
||||
as a judgment exercise), but sequenced last-but-one so Phase 8's incident to
|
||||
re-diagnose in Kibana already exists.
|
||||
**Message:** Depends on Phases 3–8 all being in place (traces, logs, metrics,
|
||||
dashboards, load-test baselines, and burn-rate alerts are the toolkit this
|
||||
phase exercises). Spoiler rules in `CLAUDE.md` apply once this phase starts.
|
||||
|
||||
**Time-box:** 1–2 evenings.
|
||||
**Time-box:** 2 evenings. This is where the whole thing pays off.
|
||||
|
||||
Babak's remaining questions are all *judgment* questions. He isn't testing whether
|
||||
you can recite what Filebeat does; he's testing whether you have a framework for
|
||||
choosing. Here's the framework, then the answers as worked examples.
|
||||
### Concepts — how to actually answer "how do I find bottlenecks?"
|
||||
|
||||
### The framework — six axes to reason on
|
||||
Babak asked how you'd *think* about this, not for a list of tools. Here's the
|
||||
method, roughly in order. The discipline is to go **top-down from the user, not
|
||||
bottom-up from the infrastructure** — bottom-up is how people spend three hours
|
||||
tuning a database that wasn't the problem.
|
||||
|
||||
Whenever someone asks "should we use X or Y" for a telemetry pipeline, work through:
|
||||
|
||||
1. **Where does the data originate?** Inside your application (you can instrument it)
|
||||
or outside it (a third-party appliance, nginx access logs, a legacy JVM you can't
|
||||
touch, syslog from a switch)? You can only use OTel SDKs where you own the code.
|
||||
2. **What shape is it already in?** Structured JSON at the source, or unstructured
|
||||
text needing grok/regex parsing? Parsing cost has to land *somewhere*.
|
||||
3. **Which signals?** Logs only, or logs + metrics + traces that need to correlate?
|
||||
4. **What's the destination, and how many?** One backend forever, or several, or one
|
||||
you expect to change?
|
||||
5. **Who pays the CPU, and where's the failure boundary?** Parsing at the edge (agent
|
||||
on every host) distributes cost and adds resilience but multiplies config surface.
|
||||
Parsing centrally concentrates cost and creates a single point of failure — but
|
||||
one place to change rules.
|
||||
6. **What already exists, and what does it cost?** Existing Elastic investment,
|
||||
existing licences, team skills, and licensing constraints (Elastic's Filebeat and
|
||||
Logstash are source-available under Elastic licensing; the OTel Collector, Fluent
|
||||
Bit and Vector are permissively licensed). Migration is never free.
|
||||
|
||||
### The answers
|
||||
|
||||
**When to use Filebeat.** When you need a lightweight shipper that tails log *files*
|
||||
from a host or container and forwards them with minimal transformation — and your
|
||||
destination is the Elastic Stack. Its strengths: tiny footprint, at-least-once
|
||||
delivery with a registry of file offsets, backpressure handling, and prebuilt modules
|
||||
for common services (nginx, MySQL, system logs) that come with parsing and dashboards
|
||||
included. Reach for it when the source is a file you don't control the format of, the
|
||||
app can't be modified, and Elasticsearch is where it's going.
|
||||
|
||||
**When to use Logstash.** When you need real *transformation*: parsing unstructured
|
||||
text with grok, enriching events with external lookups (GeoIP, a DB, a CMDB),
|
||||
splitting or merging events, conditional routing to multiple destinations, or
|
||||
buffering with a persistent on-disk queue. Logstash is a general-purpose ETL engine
|
||||
for events. It's heavyweight — a JVM — so you run a few of them centrally, not one
|
||||
per host.
|
||||
|
||||
**When to use Filebeat and Logstash together, and why.** This is the classic Elastic
|
||||
architecture and the reason is a **division of labour between edge and centre**:
|
||||
|
||||
- Filebeat runs on *every* host: cheap, tails files, handles offsets and
|
||||
backpressure, ships onward. You want the thing running in a thousand places to be
|
||||
small and boring.
|
||||
- Logstash runs on a *few* central nodes: does the expensive grok parsing,
|
||||
enrichment lookups, and fan-out to multiple sinks. Parsing rules change often, so
|
||||
you want them in a handful of places you can redeploy, not on a thousand hosts.
|
||||
- It also gives you a **buffer**: if Elasticsearch is down or slow, Logstash's
|
||||
persistent queue absorbs it and Filebeat backs off gracefully, rather than every
|
||||
host queuing independently or dropping.
|
||||
|
||||
So: Filebeat for *collection and delivery*, Logstash for *processing and routing*.
|
||||
Use both when you have many hosts **and** non-trivial parsing/enrichment **and**
|
||||
need central buffering. Skip Logstash when your logs are already structured JSON and
|
||||
Elasticsearch ingest pipelines can do the little parsing that remains.
|
||||
|
||||
**When to use OpenTelemetry.**
|
||||
|
||||
- You own the application code and want telemetry generated *from inside it* — spans
|
||||
with business context, custom metrics, logs already carrying trace IDs.
|
||||
- You need **more than one signal**, correlated. This is the strongest argument: one
|
||||
SDK, one protocol, one pipeline for traces, metrics and logs, with trace IDs
|
||||
threaded through all three. Neither Filebeat nor Logstash does traces.
|
||||
- You have a **polyglot** estate — Python, Go, TypeScript — and want one consistent
|
||||
data model across all of them.
|
||||
- You want to avoid backend lock-in. Instrument once; swap Tempo for Datadog for
|
||||
Jaeger by editing Collector YAML.
|
||||
- You need distributed tracing at all. There is no serious vendor-neutral alternative.
|
||||
- Your telemetry needs processing in flight — redaction, tail sampling, attribute
|
||||
rewriting — without touching the app.
|
||||
|
||||
**When *not* to use OpenTelemetry.** This is the question that separates people who
|
||||
have used it from people who have read about it.
|
||||
|
||||
- **You don't own the code.** A network appliance, a SaaS webhook, a vendor binary,
|
||||
an ancient app nobody will rebuild. You cannot instrument it. Ship its logs with a
|
||||
file-based collector.
|
||||
- **You only need one signal from one source with no correlation.** A single nginx
|
||||
access log going to a single log store: OTel is more machinery than the job needs.
|
||||
Fluent Bit or Filebeat is simpler and finishes the job today.
|
||||
- **You're deep in a committed ecosystem and the migration doesn't pay.** A mature
|
||||
Elastic estate with hundreds of Beats modules, dashboards and detection rules is
|
||||
years of investment. "We should use OTel because it's the standard" is not a
|
||||
business case. Note the direction of travel though: the Collector can already
|
||||
receive from and export to most of these, so a *gradual* migration is usually the
|
||||
right shape — often by putting a Collector or Vector in front of the existing
|
||||
agents rather than ripping them out.
|
||||
- **Hard resource constraints.** Embedded, edge, or extremely tight sidecar budgets,
|
||||
where a full SDK plus Collector is too much. Fluent Bit is measured in megabytes.
|
||||
- **The signal is genuinely business analytics, not telemetry.** Clickstream data for
|
||||
a data warehouse belongs in a data pipeline, not a telemetry pipeline. Different
|
||||
retention, different query patterns, different cost model.
|
||||
- **Maturity gaps for a specific need in a specific language.** OTel logs support has
|
||||
matured at different rates per language, and profiling is still alpha. Check the
|
||||
status page for your language before betting on a signal.
|
||||
- **Team capacity.** OTel has real conceptual surface area — you've just spent weeks
|
||||
on it. A two-person team that needs error tracking by Friday should install Sentry.
|
||||
|
||||
The senior framing to give Babak: *these tools aren't competitors so much as
|
||||
different layers.* OpenTelemetry is a **generation and transport standard** for
|
||||
telemetry you own. Filebeat and Fluent Bit are **collectors for telemetry you
|
||||
don't**. Logstash and Vector are **processing engines**. Loki, Tempo, Prometheus and
|
||||
Elasticsearch are **stores**. Most real architectures use one from each layer, and
|
||||
the interesting question is always which layer a given problem lives in.
|
||||
1. **Start at the SLI, not the server.** Which endpoint, which percentile, which
|
||||
window, compared to what baseline? "It's slow" is not a starting point. "p95 of
|
||||
`POST /api/transactions` went from 180 ms to 2.1 s at 14:05" is.
|
||||
2. **Localise to a service** using RED + the service graph. Which node's duration
|
||||
grew? Note that a slow *downstream* makes its *upstream* look slow too, so read
|
||||
the graph from the leaves inward.
|
||||
3. **Get to exemplar traces.** Click the latency spike through to a trace, or query
|
||||
TraceQL directly (`{ duration > 2s }`, `{ span.status = error }`). You want
|
||||
several slow traces plus one fast trace of the same endpoint as a control.
|
||||
4. **Read the trace for five specific shapes:**
|
||||
- **A long span** — the work itself is slow. Look at its attributes: which query,
|
||||
which host, which parameters.
|
||||
- **A gap** — wall-clock time not covered by any span. This is the interesting
|
||||
one. Gaps mean untraced work: queue wait, connection-pool checkout, GC, a
|
||||
blocked event loop, or a missing instrumentation library. Ledgerline's
|
||||
`pool_starvation` and `blocking_event_loop` both show up as gaps.
|
||||
- **Serialisation** — spans laid end to end that could have been concurrent.
|
||||
Ledgerline's `no_batching` looks exactly like this.
|
||||
- **Repetition** — the same span name 150 times in a row. That's your N+1.
|
||||
- **Retries** — the same client span repeated 3–4 times with errors. That's
|
||||
`retry_storm`, and note that it makes an *overload* worse, never better.
|
||||
5. **Separate latency from saturation.** Switch from RED to USE on the suspected
|
||||
resource: connection pool in-use vs pool size, stream consumer lag, CPU,
|
||||
memory. A resource at 100% utilisation with a growing queue is saturated;
|
||||
latency there is a *symptom*, not the cause.
|
||||
6. **Use Little's Law to reason about capacity.** `L = λ × W`: required concurrency
|
||||
equals arrival rate × latency. At 50 req/s with 200 ms latency you need 10
|
||||
concurrent slots. If your pool is 5, you queue — and queueing latency grows
|
||||
non-linearly as you approach saturation. This one formula explains most
|
||||
connection-pool and worker-count incidents you'll ever see.
|
||||
7. **Check for tail amplification** (Phase 7) before blaming any single service.
|
||||
8. **Form one hypothesis and test it with a controlled change.** Toggle exactly one
|
||||
chaos flag, re-run the *identical* k6 profile, and compare the *same percentile*
|
||||
over the *same duration*. One variable at a time. Never declare a fix from a
|
||||
dashboard that "looks better."
|
||||
9. **Expect the bottleneck to move.** Fix the slowest thing and the second-slowest
|
||||
becomes the bottleneck. Re-measure after every change; stop when you're inside
|
||||
the SLO, not when you run out of ideas.
|
||||
10. **Know what your tools can't see.** Traces won't show you CPU time inside a span,
|
||||
or lock waits inside Postgres, or which line of Python burned the cycles. That's
|
||||
where DB-side tools (`pg_stat_statements`, `EXPLAIN ANALYZE`) and continuous
|
||||
profiling (Pyroscope, bundled in `grafana/otel-lgtm`) take over.
|
||||
|
||||
### Resources
|
||||
|
||||
- Better Stack — *Filebeat vs Logstash: How to Choose* (free article).
|
||||
- SigNoz — *OpenTelemetry vs ELK Stack* (free article).
|
||||
- OpenTelemetry docs → Collector → *filelog receiver* and *transform processor* —
|
||||
read these to see how the Collector does the Filebeat/Logstash job itself.
|
||||
- OpenTelemetry docs → *Instrumentation status by language* — for the maturity
|
||||
caveats.
|
||||
- Brendan Gregg — *The USE Method* (free article). Short, and the canonical framing
|
||||
for saturation.
|
||||
- Tom Wilkie / Grafana — *The RED Method* (free article + conference talk).
|
||||
- *The Tail at Scale* (Dean & Barroso, CACM) — free PDF, ~8 pages, and the best
|
||||
thing ever written on why microservice latency behaves the way it does.
|
||||
- Grafana docs → Tempo → **TraceQL**. You'll live in this.
|
||||
- Postgres: `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` — you already know
|
||||
these; connect them to the trace's DB spans.
|
||||
|
||||
### Build — and this is no longer hypothetical
|
||||
### Build
|
||||
|
||||
Your platform repo ships a **complete second stack**: `playbooks/elk-grafana.yml`
|
||||
deploys Filebeat → Logstash → Elasticsearch → Kibana/Grafana on the same host, with
|
||||
Filebeat reading Docker container logs and Logstash parsing JSON into daily
|
||||
`logs-YYYY.MM.dd` indices. That is almost certainly *why* Babak asked these
|
||||
questions — the codebase he handed you contains both worlds.
|
||||
1. `make chaos-random` (a random 3–6 flags, seed never logged). **Do not look at
|
||||
`docs/PATHOLOGIES.md`.**
|
||||
2. Run `make load-average` for 10 minutes.
|
||||
3. Hunt. Write `docs/INVESTIGATION.md` as you go — timestamped, hypothesis-driven,
|
||||
like a real incident doc: what you observed, what you hypothesised, what you
|
||||
checked, what it turned out to be. Include screenshots.
|
||||
4. When you think you've found them all, run `make chaos-clean`, enable them one at
|
||||
a time, and confirm each signature.
|
||||
5. *Then* read `docs/PATHOLOGIES.md` and score yourself.
|
||||
6. Repeat once with a new random set. The second run is where the method actually
|
||||
becomes yours.
|
||||
|
||||
1. `ansible-playbook playbooks/elk-grafana.yml`. Now the **same Ledgerline logs**
|
||||
flow down two pipelines: Filebeat → Logstash → Elasticsearch, and
|
||||
Alloy → Loki. Compare them on the same data.
|
||||
2. Test the direct ingest path the README documents — pipe a JSON line into
|
||||
Logstash's TCP input on 5000, then query Elasticsearch — so you understand where
|
||||
parsing happens before you change it.
|
||||
3. **Keep a copy of the Phase 0 plain-text logs** and write a Logstash `grok` filter
|
||||
that parses them. Then do the same job with Alloy's `loki.process` stages. Time
|
||||
both. Note which one you'd rather debug at 2 a.m.
|
||||
4. Compare the *investigation experience* directly: take one incident you already
|
||||
diagnosed in Phase 8 and re-run it in Kibana Discover. What can you do there that
|
||||
you can't in Grafana Explore, and vice versa? The honest answer is not "Grafana
|
||||
wins" — Elasticsearch's full-text search and aggregations genuinely beat LogQL
|
||||
for some questions, and Loki's cost model genuinely beats Elasticsearch's for
|
||||
others.
|
||||
5. Note the licensing axis for real: Filebeat and Logstash ship under Elastic
|
||||
licensing, while the OpenTelemetry Collector, Alloy, Fluent Bit and Vector are
|
||||
permissively licensed. For some employers that alone decides it.
|
||||
6. Write `docs/PIPELINE-CHOICES.md`: the six axes, the answers above, and — most
|
||||
valuably — what you actually found annoying about each. Having run both stacks
|
||||
against the same application is the difference between reciting a comparison and
|
||||
arguing one.
|
||||
**Done when:** you find at least 4 of the enabled pathologies from telemetry alone,
|
||||
and `docs/INVESTIGATION.md` reads like something you'd hand to a colleague.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user