Add deployment phase and Traefik compose override
Insert a new Phase 1 (deployment: Compose + Traefik on a server, marked Done) between the base project and the OTel concepts phase, and shift the old phases 1-10 to 2-11. Update summary, intro, README and CLAUDE.md references (spoiler rules now target Phase 9, the bottleneck hunt). Add docker-compose.traefik.yml: an override that fronts the gateway with Traefik and Let's Encrypt and removes all other published host ports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Movses Aslanyan
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
b2112e55e1
commit
5e514ecf25
+64
-79
@@ -1,89 +1,74 @@
|
||||
# Phase 1 — The model, and one signal end to end
|
||||
# Phase 1 — Deployment: Ledgerline on a server behind Traefik
|
||||
|
||||
*Part of `docs/roadmap/` — see [`intro.md`](intro.md) for platform/background context and [`summary.md`](summary.md) for how every phase fits together. Prev: [`phase-0.md`](phase-0.md) · Next: [`phase-2.md`](phase-2.md)*
|
||||
|
||||
**Status:** Not started
|
||||
**Status:** Done
|
||||
|
||||
**Message:** Blocked on nothing — Phase 0 is done and the platform is
|
||||
described above. Waiting to begin.
|
||||
**Message:** Ledgerline is running on a remote server and served over HTTPS at
|
||||
**https://ledgerline.vione.cloud**. Verified against the live deployment, not
|
||||
just the config: the frontend loads, `/api/health/all` reports every service
|
||||
`ok`, and the origin serves a real Let's Encrypt certificate (valid until
|
||||
2027-01-07, auto-renewed by Traefik).
|
||||
|
||||
**Time-box:** one evening.
|
||||
- **Target:** the throwaway server `czy` (Ubuntu 24.04, 1 vCPU, 1 GB RAM), repo
|
||||
at `/opt/apps/LedgerLine`. Docker 29.9 and Compose v5.6 installed from
|
||||
Docker's apt repo. A 2 GB swapfile was added (persisted in `/etc/fstab`) —
|
||||
nine containers do not fit in 1 GB of RAM without it, and steady state still
|
||||
uses roughly 700 MB plus swap.
|
||||
- **How it's wired:** `docker-compose.traefik.yml` is an *override*, not an
|
||||
edit of `docker-compose.yml`, so local development is unchanged. It adds
|
||||
Traefik v3.6 on ports 80/443 (Docker provider, `exposedByDefault=false`,
|
||||
HTTP→HTTPS redirect, Let's Encrypt via HTTP-01), routes
|
||||
``Host(`ledgerline.vione.cloud`)`` to the gateway on port 8000, and strips
|
||||
the published host ports from every other service (`ports: !reset []`) so
|
||||
Postgres, Redis and the services are reachable only on the Compose network.
|
||||
- **No application or `tests/` changes.** The Dockerfile and base compose file
|
||||
are untouched; schema applied and data seeded on the server with the same
|
||||
steps as `make seed` (800 accounts, 250,000 transactions).
|
||||
- **Deploy / redeploy command** (always both files — a bare `docker compose up`
|
||||
would use the base file alone and publish every port):
|
||||
|
||||
**Why:** you need the vocabulary, but more importantly you need to see one trace
|
||||
appear in Grafana within an hour, so the rest of the roadmap feels concrete rather
|
||||
than abstract.
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --build
|
||||
```
|
||||
|
||||
- **Cloudflare gotchas hit on the way:**
|
||||
- The first certificate attempt failed with an ACME 404 because the
|
||||
Cloudflare DNS record did not yet point at the server — a packet capture
|
||||
on the server showed no inbound traffic from Cloudflare at all. Traefik
|
||||
only retries issuance on restart, so after fixing the record it needed
|
||||
`docker compose ... restart traefik`.
|
||||
- With the orange-cloud proxy on, the SSL/TLS mode must be **Full** —
|
||||
**Flexible** combined with Traefik's HTTPS redirect causes a redirect loop.
|
||||
- **Caveats to remember:** the deployment is public, including the
|
||||
`/api/admin/chaos` endpoints (anyone who finds them can flip flags);
|
||||
internet scanners probed the site within seconds of it going live. The
|
||||
server is disposable by design, and Traefik's state lives in the
|
||||
`letsencrypt` volume, which is lost on an OS reinstall (a new certificate is
|
||||
simply issued on the next run).
|
||||
|
||||
**Time-box:** 1 evening.
|
||||
|
||||
**Deliverable:** the unmodified-behaviour Ledgerline system reachable at a real
|
||||
domain over HTTPS, reproducible with one compose command.
|
||||
|
||||
### Concepts
|
||||
|
||||
- **Monitoring vs observability.** Monitoring answers questions you knew to ask
|
||||
(known unknowns): dashboards and alerts built from predefined aggregates.
|
||||
Observability is a property of the system: can you ask a *new* question about
|
||||
production, that you never anticipated, without shipping new code? Charity Majors'
|
||||
framing — monitoring is known-unknowns, observability is unknown-unknowns. The
|
||||
practical implication for you: emit **wide, high-dimensional events** (rich span
|
||||
attributes), not just pre-aggregated counters.
|
||||
- **The OTel architecture split: API vs SDK vs Collector.**
|
||||
- The **API** is what your code and third-party instrumentation libraries call.
|
||||
If no SDK is configured, API calls are no-ops — that's deliberate, it means a
|
||||
library can be instrumented without forcing a dependency on any implementation.
|
||||
- The **SDK** is the engine: sampling, batching, context propagation, export.
|
||||
Application owners configure the SDK; library authors only touch the API.
|
||||
- The **Collector** is a standalone process that receives, processes and exports
|
||||
telemetry. Optional, but you should use one.
|
||||
- **OTLP** is the wire protocol — gRPC on 4317, HTTP/protobuf on 4318.
|
||||
- **Resource** vs **attributes**: a Resource describes the *producer* (`service.name`,
|
||||
`service.version`, `deployment.environment`) and is attached to every signal from
|
||||
that process. Attributes describe the individual span/metric/log.
|
||||
- **Semantic conventions**: standardised attribute names (`http.request.method`,
|
||||
`db.system`, `server.address`). Use them for anything standard; namespace your own
|
||||
(`ledgerline.account.id`). This is what makes cross-service, cross-language
|
||||
correlation work — and it's what Grafana's prebuilt dashboards key off.
|
||||
- **Signals**: traces, metrics, logs. (Profiling is a fourth signal, in alpha —
|
||||
aware of, not learning yet.)
|
||||
- **Why a Collector rather than exporting straight to Tempo/Loki/Prometheus:** your
|
||||
app learns one endpoint. Swapping backends becomes a YAML edit, not a redeploy of
|
||||
seven services. The Collector also does the things you don't want in your app:
|
||||
retry, PII redaction, tail sampling, memory limiting, fan-out to multiple
|
||||
destinations.
|
||||
|
||||
### Resources
|
||||
|
||||
- freeCodeCamp *OpenTelemetry Course* (~1 h, free) — watch at 1.5×, take notes on the
|
||||
M.E.L.T. and context-propagation sections; skip the Node code.
|
||||
- OpenTelemetry docs → Concepts → *Components* and *Signals*. ~30 min.
|
||||
- Optional: run the OpenTelemetry Demo (`docker compose up` in the astronomy-shop
|
||||
repo) once and click around Grafana/Jaeger for 20 minutes. Don't try to understand
|
||||
it — just see what "done" looks like.
|
||||
|
||||
### Build
|
||||
|
||||
1. **Deploy the platform.** Provision a VM, point `inventory/hosts.yml` at it,
|
||||
`ansible-galaxy collection install -r requirements.yml`, change
|
||||
`grafana_admin_password` (vault it), then
|
||||
`ansible-playbook playbooks/grafana-stack.yml`. Verify every endpoint from the
|
||||
README's table answers: Grafana 3000, Prometheus 9090, Alertmanager 9093,
|
||||
Loki 3100, Tempo 3200, Alloy UI 12345, OTLP 4317/4318.
|
||||
Then open the Alloy UI and **read the component graph** — it shows you the exact
|
||||
pipeline, including the fact that the OTLP receiver has only a traces output.
|
||||
Seeing that in the UI before you get bitten by it is worth ten minutes.
|
||||
*(Optional laptop fallback while the VM is being set up: `grafana/otel-lgtm`,
|
||||
a single container with everything pre-wired. Useful for a first trace, but
|
||||
don't build on it — its pipeline differs from your real platform in exactly the
|
||||
ways that matter.)*
|
||||
2. Pick the **`sanctions`** service (the smallest) and add nothing but zero-code
|
||||
auto-instrumentation:
|
||||
```
|
||||
pip install opentelemetry-distro opentelemetry-exporter-otlp
|
||||
opentelemetry-bootstrap -a install
|
||||
```
|
||||
and change its command to
|
||||
`opentelemetry-instrument uvicorn services.sanctions.app:app --host 0.0.0.0 --port 8003`
|
||||
with `OTEL_SERVICE_NAME=sanctions`,
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://<obs-host>:4318`,
|
||||
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` — pointing at **Alloy**, not at
|
||||
Tempo directly.
|
||||
3. Fire a request. Open Grafana → Explore → Tempo → find the trace.
|
||||
|
||||
**Done when:** you can see a single-service trace in Tempo, and you can explain out
|
||||
loud what the Resource attributes on it mean and where they came from.
|
||||
- **Traefik's model:** *entrypoints* (the ports it listens on), *routers*
|
||||
(rule → service), *services* (where to send traffic), *middlewares*, and
|
||||
*providers* (where config comes from — here, Docker labels on containers).
|
||||
Compare with nginx, where the same information lives in one static config
|
||||
file.
|
||||
- **Compose override files.** `-f base -f override` merges the files;
|
||||
`!reset` clears an inherited key (here, `ports`) instead of appending to it.
|
||||
- **ACME HTTP-01 behind a CDN.** Let's Encrypt must be able to reach port 80 on
|
||||
the *hostname's actual origin*. With Cloudflare proxying, that only works
|
||||
once the DNS record points at the right server.
|
||||
- **Why this phase exists before observability.** The telemetry work in
|
||||
Phases 2–11 is more convincing when the system under observation is a real
|
||||
deployment rather than a laptop, and the platform (`observability-ansible`)
|
||||
is already a remote deployment — Ledgerline now sits on the same footing.
|
||||
|
||||
**Done when:** `https://ledgerline.vione.cloud` shows the frontend with a valid
|
||||
certificate; `/api/health/all` is all `ok`; only ports 22, 80 and 443 are
|
||||
reachable on the server; and the deploy is one documented compose command.
|
||||
|
||||
Reference in New Issue
Block a user