Add observability platform Ansible project

Deploys the Grafana stack (Prometheus/Alertmanager/Loki/Tempo/Grafana with Alloy as collector) and the ELK stack (Filebeat/Logstash/Elasticsearch/ Kibana) via Docker Compose on a target host. External to the application repo, pre-built by the mentor.
2026-09-05 08:11:34 +03:30
parent eb99cdbb8c
commit bffc42c9d3
26 changed files with 1056 additions and 0 deletions
@@ -0,0 +1,204 @@
+# Deploying Two Observability Stacks with Ansible
+
+This project deploys two independent stacks on a single Ubuntu/Debian host using Docker Compose:
+
+1. **ELK + Grafana:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana
+2. **Grafana Observability:** Prometheus + Alertmanager + Loki + Tempo + Grafana, along with Alloy and Node Exporter for data collection
+
+> The default settings are suitable for Lab, Demo, and training purposes. For Production, be sure to read the "Production Notes" section.
+
+## Operational Diagrams
+
+- `diagrams/01-elk-grafana-operational.png` and its SVG vector version
+- `diagrams/02-grafana-lgtm-operational.png` and its SVG vector version
+
+Path colors in the second diagram:
+
+- Green: Metrics
+- Blue: Logs
+- Purple: Traces
+- Dashed orange: Query/Response
+- Red: Alert delivery
+
+## Project Structure
+
+```text
+inventory/hosts.yml                 SSH targets
+group_vars/all.yml                  Versions, Grafana password, and variables
+playbooks/elk-grafana.yml           Installs ELK + Grafana
+playbooks/grafana-stack.yml         Installs Grafana/Prometheus/Loki/Tempo
+roles/docker/                       Installs Docker Engine and Compose v2
+roles/observability/                Templates, execution, and health checks
+diagrams/                           PNG and SVG diagrams
+```
+
+## Prerequisites
+
+- Control node: Ansible Core 2.16 or newer
+- Target: Ubuntu 22.04/24.04 or Debian 12, with SSH and sudo access
+- Recommended minimum for the ELK Lab: 4 vCPUs, 8 GB RAM, and 50 GB disk
+- Recommended minimum for the Grafana Stack Lab: 2 vCPUs, 4 GB RAM, and 30 GB disk
+
+On the control node:
+
+```bash
+cd observability-ansible
+ansible-galaxy collection install -r requirements.yml
+```
+
+## 1. Configure the Inventory
+
+Edit the `inventory/hosts.yml` file:
+
+```yaml
+all:
+  children:
+    observability:
+      hosts:
+        obs01:
+          ansible_host: 10.10.20.15
+          ansible_user: ubuntu
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519
+```
+
+Test the connection:
+
+```bash
+ansible observability -m ping
+```
+
+## 2. Configure Variables and Secrets
+
+Versions and core settings live in `group_vars/all.yml`. Be sure to change the default Grafana password. To store the password securely:
+
+```bash
+ansible-vault encrypt_string 'A-strong-password' --name grafana_admin_password
+```
+
+Replace the `grafana_admin_password` value with the Vault output, and run the playbook with `--ask-vault-pass`.
+
+If your application exposes a `/metrics` endpoint, add it here:
+
+```yaml
+prometheus_extra_scrape_jobs:
+  - job_name: checkout-api
+    targets:
+      - 10.20.0.15:8080
+      - 10.20.0.16:8080
+```
+
+To send alerts to a webhook, also set the following value:
+
+```yaml
+alertmanager_webhook_url: "https://example.internal/alerts"
+```
+
+## 3. Install ELK + Grafana
+
+```bash
+ansible-playbook playbooks/elk-grafana.yml
+```
+
+Endpoints:
+
+| Service | URL/Port | Purpose |
+| --- | --- | --- |
+| Elasticsearch | `http://SERVER:9200` | Index and Search API |
+| Logstash Beats input | `SERVER:5044` | Receives Filebeat |
+| Logstash JSON/TCP | `SERVER:5000` | Testing and direct JSON Lines ingest |
+| Kibana | `http://SERVER:5601` | Discover and Elastic analysis |
+| Grafana | `http://SERVER:3000` | Dashboards on top of Elasticsearch |
+
+Filebeat in this project reads the Docker container logs on the same host and sends them to Logstash. Logstash parses the JSON and writes it into the daily `logs-YYYY.MM.dd` index.
+
+Test ingest:
+
+```bash
+printf '%s\n' '{"message":"checkout completed","level":"info","service":"checkout"}' \
+  | nc SERVER 5000
+
+curl 'http://SERVER:9200/logs-*/_search?pretty'
+```
+
+## 4. Install Grafana + Prometheus + Alertmanager + Loki + Tempo
+
+```bash
+ansible-playbook playbooks/grafana-stack.yml
+```
+
+Endpoints:
+
+| Service | URL/Port | Purpose |
+| --- | --- | --- |
+| Grafana | `http://SERVER:3000` | Explore, Dashboards, and correlation |
+| Prometheus | `http://SERVER:9090` | Metrics, PromQL, and alert rules |
+| Alertmanager | `http://SERVER:9093` | Grouping, deduplication, and routing of alerts |
+| Loki | `http://SERVER:3100` | Log ingestion/query and LogQL |
+| Tempo | `http://SERVER:3200` | Trace query and TraceQL |
+| Alloy UI | `http://SERVER:12345` | Collector pipeline status |
+| OTLP/gRPC | `SERVER:4317` | Receives traces from the application |
+| OTLP/HTTP | `SERVER:4318` | Receives traces from the application |
+| Node Exporter | `http://SERVER:9100/metrics` | Host-level metrics |
+
+The Prometheus, Loki, Tempo, and Alertmanager data sources are automatically provisioned in Grafana. Grafana uses exemplars for Metrics → Trace, `trace_id` for Logs → Trace, and Prometheus for the Tempo service graph.
+
+Example OpenTelemetry environment variables for your application:
+
+```bash
+export OTEL_SERVICE_NAME=checkout-api
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://SERVER:4318
+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+```
+
+Alloy pushes Docker container logs to Loki and forwards OTLP traces to Tempo. Prometheus scrapes internal services and Node Exporter. The sample rules `HostExporterDown` and `HostHighCPU` are also installed.
+
+## Day-to-Day Operations
+
+On the target server:
+
+```bash
+cd /opt/observability/elk-grafana
+docker compose ps
+docker compose logs -f --tail=100
+
+cd /opt/observability/grafana-stack
+docker compose ps
+docker compose logs -f --tail=100
+```
+
+Re-running the playbook is idempotent; changing a template or version tag triggers a recreate/restart of the relevant stack. Data is kept in Docker named volumes.
+
+## Removing a Stack
+
+To stop and remove the containers without deleting the data:
+
+```bash
+cd /opt/observability/STACK_NAME
+docker compose down
+```
+
+The `-v` flag deletes all volumes and monitoring data; only use it when you have a backup and specifically intend to do this.
+
+## Production Notes
+
+The default playbook intentionally builds a single-node, easy-to-understand deployment. For Production:
+
+- Enable TLS, authentication, and RBAC on all endpoints; don't expose storage ports directly to the internet.
+- Keep secrets in Ansible Vault or a Secret Manager.
+- Run Elasticsearch with at least three nodes, a proper snapshot repository, and ILM.
+- Use S3-compatible object storage and replication for Loki and Tempo.
+- Design Prometheus with remote-write/Thanos/Mimir, retention, and adequate disk capacity.
+- Run Alertmanager in HA and test your notification routes.
+- Add a reverse proxy, DNS, certificates, firewall rules, and backup/restore testing.
+- Upgrade versions in staging first; they are intentionally pinned in `group_vars/all.yml`.
+
+## Official Resources
+
+- [Elastic Docker deployment](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-docker-compose)
+- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
+- [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
+- [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/)
+- [Grafana Loki configuration](https://grafana.com/docs/loki/latest/configure/)
+- [Grafana Tempo configuration](https://grafana.com/docs/tempo/latest/configuration/)
+- [Grafana Alloy](https://grafana.com/docs/alloy/latest/)
+- [Ansible Docker Compose v2 module](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_module.html)
@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventory/hosts.yml
+roles_path = roles
+host_key_checking = False
+interpreter_python = auto_silent
+retry_files_enabled = False
+
+[ssh_connection]
+pipelining = True
+
@@ -0,0 +1,44 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080" viewBox="0 0 1920 1080">
+  <defs>
+    <linearGradient id="bg" x1="0" y1="0" x2="1" y2="1"><stop stop-color="#07111f"/><stop offset="1" stop-color="#111a2d"/></linearGradient>
+    <filter id="shadow"><feDropShadow dx="0" dy="10" stdDeviation="12" flood-opacity=".25"/></filter>
+    <marker id="blueArrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#35b9ff"/></marker>
+    <marker id="orangeArrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#ff9f43"/></marker>
+    <marker id="redArrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#ff5f6d"/></marker>
+    <style>
+      text{font-family:Inter,Segoe UI,Arial,sans-serif;fill:#eaf2ff}.title{font-size:42px;font-weight:800}.sub{font-size:18px;fill:#9fb0c9}.layer{font-size:15px;font-weight:700;letter-spacing:2px;fill:#7185a3}.box{fill:#121f35;stroke:#314762;stroke-width:2;rx:18;filter:url(#shadow)}.boxtitle{font-size:23px;font-weight:750}.meta{font-size:15px;fill:#afbdd1}.pill{fill:#1d304d;stroke:#405a7a}.pilltext{font-size:14px;fill:#d7e6fb}.flow{fill:none;stroke:#35b9ff;stroke-width:4;marker-end:url(#blueArrow)}.query{fill:none;stroke:#ff9f43;stroke-width:4;stroke-dasharray:12 9;marker-start:url(#orangeArrow);marker-end:url(#orangeArrow)}.alert{fill:none;stroke:#ff5f6d;stroke-width:4;marker-end:url(#redArrow)}.label{font-size:14px;font-weight:650;paint-order:stroke;stroke:#0a1424;stroke-width:7px;stroke-linejoin:round}.stage{fill:#0b1728;stroke:#223651;stroke-width:1.5;rx:22}
+    </style>
+  </defs>
+  <rect width="1920" height="1080" fill="url(#bg)"/>
+  <text x="70" y="74" class="title">ELK + Grafana — Operational Data Flow</text>
+  <text x="70" y="108" class="sub">Collection → processing → indexed storage → search, dashboards and alert delivery</text>
+
+  <rect x="45" y="150" width="295" height="750" class="stage"/><text x="70" y="186" class="layer">DATA SOURCES</text>
+  <rect x="365" y="150" width="285" height="750" class="stage"/><text x="390" y="186" class="layer">COLLECTION</text>
+  <rect x="675" y="150" width="285" height="750" class="stage"/><text x="700" y="186" class="layer">PROCESSING</text>
+  <rect x="985" y="150" width="390" height="750" class="stage"/><text x="1010" y="186" class="layer">INDEXED STORAGE</text>
+  <rect x="1400" y="150" width="475" height="750" class="stage"/><text x="1425" y="186" class="layer">QUERY, VISUALIZE &amp; ALERT</text>
+
+  <g><rect x="70" y="235" width="245" height="130" class="box"/><circle cx="108" cy="277" r="22" fill="#4ea1ff"/><text x="101" y="285" font-size="24" font-weight="800">A</text><text x="145" y="278" class="boxtitle">Applications</text><text x="95" y="320" class="meta">JSON / text logs</text><text x="95" y="344" class="meta">trace_id in log context</text></g>
+  <g><rect x="70" y="430" width="245" height="130" class="box"/><circle cx="108" cy="472" r="22" fill="#6ad36a"/><text x="101" y="480" font-size="23" font-weight="800">S</text><text x="145" y="473" class="boxtitle">System logs</text><text x="95" y="515" class="meta">journald / syslog</text><text x="95" y="539" class="meta">auth and kernel events</text></g>
+  <g><rect x="70" y="625" width="245" height="130" class="box"/><circle cx="108" cy="667" r="22" fill="#9c7cff"/><text x="101" y="675" font-size="23" font-weight="800">C</text><text x="145" y="668" class="boxtitle">Containers</text><text x="95" y="710" class="meta">Docker JSON logs</text><text x="95" y="734" class="meta">Kubernetes log files</text></g>
+
+  <g><rect x="395" y="340" width="225" height="250" class="box" stroke="#36c5f0"/><circle cx="507" cy="405" r="38" fill="#16769a"/><text x="493" y="417" font-size="36" font-weight="850">F</text><text x="507" y="472" class="boxtitle" text-anchor="middle">Filebeat</text><text x="507" y="504" class="meta" text-anchor="middle">tail • multiline • metadata</text><rect x="430" y="530" width="155" height="34" rx="17" class="pill"/><text x="507" y="552" class="pilltext" text-anchor="middle">disk registry</text></g>
+  <g><rect x="705" y="340" width="225" height="250" class="box" stroke="#f5c142"/><circle cx="817" cy="405" r="38" fill="#876b10"/><text x="804" y="417" font-size="34" font-weight="850">L</text><text x="817" y="472" class="boxtitle" text-anchor="middle">Logstash</text><text x="817" y="504" class="meta" text-anchor="middle">parse • enrich • route</text><rect x="740" y="530" width="155" height="34" rx="17" class="pill"/><text x="817" y="552" class="pilltext" text-anchor="middle">persistent queue</text></g>
+
+  <g><rect x="1025" y="275" width="310" height="390" class="box" stroke="#31b8dc"/><text x="1180" y="325" class="boxtitle" text-anchor="middle">Elasticsearch cluster</text><circle cx="1110" cy="420" r="66" fill="#123c58" stroke="#43c5e8" stroke-width="4"/><circle cx="1248" cy="420" r="66" fill="#123c58" stroke="#43c5e8" stroke-width="4"/><circle cx="1179" cy="540" r="66" fill="#123c58" stroke="#43c5e8" stroke-width="4"/><text x="1110" y="413" font-size="20" text-anchor="middle" font-weight="750">data-1</text><text x="1110" y="440" class="meta" text-anchor="middle">primary</text><text x="1248" y="413" font-size="20" text-anchor="middle" font-weight="750">data-2</text><text x="1248" y="440" class="meta" text-anchor="middle">replica</text><text x="1179" y="533" font-size="20" text-anchor="middle" font-weight="750">data-3</text><text x="1179" y="560" class="meta" text-anchor="middle">replica</text><rect x="1080" y="610" width="200" height="34" rx="17" class="pill"/><text x="1180" y="632" class="pilltext" text-anchor="middle">data streams • ILM</text></g>
+
+  <g><rect x="1435" y="235" width="190" height="150" class="box" stroke="#e758a8"/><text x="1530" y="285" class="boxtitle" text-anchor="middle">Kibana</text><text x="1530" y="322" class="meta" text-anchor="middle">Discover / Lens</text><rect x="1470" y="340" width="120" height="30" rx="15" class="pill"/><text x="1530" y="361" class="pilltext" text-anchor="middle">TCP 5601</text></g>
+  <g><rect x="1660" y="235" width="190" height="150" class="box" stroke="#ff9830"/><text x="1755" y="285" class="boxtitle" text-anchor="middle">Grafana</text><text x="1755" y="322" class="meta" text-anchor="middle">Explore / dashboards</text><rect x="1695" y="340" width="120" height="30" rx="15" class="pill"/><text x="1755" y="361" class="pilltext" text-anchor="middle">TCP 3000</text></g>
+  <g><rect x="1495" y="625" width="295" height="150" class="box" stroke="#ff5f6d"/><text x="1642" y="675" class="boxtitle" text-anchor="middle">Notification channels</text><text x="1642" y="712" class="meta" text-anchor="middle">Slack • Email • PagerDuty</text><rect x="1560" y="730" width="165" height="30" rx="15" class="pill"/><text x="1642" y="751" class="pilltext" text-anchor="middle">webhook / SMTP</text></g>
+
+  <path d="M315 300 C355 300 355 385 395 385" class="flow"/><path d="M315 495 H395" class="flow"/><path d="M315 690 C355 690 355 545 395 545" class="flow"/>
+  <path d="M620 465 H705" class="flow"/><text x="662" y="442" class="label" text-anchor="middle">Beats/TCP</text><text x="662" y="462" class="label" text-anchor="middle">5044</text>
+  <path d="M930 465 H1025" class="flow"/><text x="977" y="425" class="label" text-anchor="middle">Bulk API</text><text x="977" y="445" class="label" text-anchor="middle">HTTP(S) 9200</text>
+  <path d="M1335 355 C1385 355 1385 310 1435 310" class="query"/><path d="M1335 500 C1450 500 1535 310 1660 310" class="query"/><text x="1393" y="482" class="label" text-anchor="middle">Search API :9200</text>
+  <path d="M1755 385 C1755 505 1680 545 1642 625" class="alert"/><path d="M1530 385 C1530 505 1580 550 1615 625" class="alert"/><text x="1720" y="555" class="label" text-anchor="middle">alert rules</text>
+
+  <rect x="70" y="945" width="1780" height="80" rx="18" fill="#0c1728" stroke="#263b58"/>
+  <line x1="110" y1="985" x2="180" y2="985" class="flow"/><text x="200" y="991" class="meta">ingest / write path</text><line x1="465" y1="985" x2="535" y2="985" class="query"/><text x="555" y="991" class="meta">query + response</text><line x1="830" y1="985" x2="900" y2="985" class="alert"/><text x="920" y="991" class="meta">alert delivery</text><text x="1270" y="991" class="meta">Production: enable TLS/RBAC, snapshots, ≥3 ES nodes and external secrets</text>
+</svg>
+
@@ -0,0 +1,55 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080" viewBox="0 0 1920 1080">
+  <defs>
+    <linearGradient id="bg2" x1="0" y1="0" x2="1" y2="1"><stop stop-color="#08101d"/><stop offset="1" stop-color="#171326"/></linearGradient>
+    <filter id="shadow2"><feDropShadow dx="0" dy="10" stdDeviation="12" flood-opacity=".25"/></filter>
+    <marker id="mBlue" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#47b7ff"/></marker>
+    <marker id="mGreen" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#55d187"/></marker>
+    <marker id="mPurple" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#a987ff"/></marker>
+    <marker id="mOrange" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#ff9a3d"/></marker>
+    <marker id="mRed" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#ff5f6d"/></marker>
+    <style>
+      text{font-family:Inter,Segoe UI,Arial,sans-serif;fill:#edf4ff}.title{font-size:42px;font-weight:800}.sub{font-size:18px;fill:#9fb0c9}.layer{font-size:15px;font-weight:700;letter-spacing:2px;fill:#7185a3}.stage{fill:#0c1626;stroke:#273a56;stroke-width:1.5;rx:22}.box{fill:#142138;stroke:#324967;stroke-width:2;rx:18;filter:url(#shadow2)}.boxtitle{font-size:23px;font-weight:750}.meta{font-size:15px;fill:#b2c0d3}.pill{fill:#1e304b;stroke:#3d5878}.pilltext{font-size:14px;fill:#d9e8fb}.metric{fill:none;stroke:#55d187;stroke-width:4;marker-end:url(#mGreen)}.log{fill:none;stroke:#47b7ff;stroke-width:4;marker-end:url(#mBlue)}.trace{fill:none;stroke:#a987ff;stroke-width:4;marker-end:url(#mPurple)}.query{fill:none;stroke:#ff9a3d;stroke-width:4;stroke-dasharray:12 9;marker-start:url(#mOrange);marker-end:url(#mOrange)}.alert{fill:none;stroke:#ff5f6d;stroke-width:4;marker-end:url(#mRed)}.label{font-size:14px;font-weight:650;paint-order:stroke;stroke:#0a1424;stroke-width:7px;stroke-linejoin:round}
+    </style>
+  </defs>
+  <rect width="1920" height="1080" fill="url(#bg2)"/>
+  <text x="70" y="74" class="title">Grafana Observability Stack — Operational Data Flow</text>
+  <text x="70" y="108" class="sub">Metrics (Prometheus) • logs (Loki) • traces (Tempo) • alert routing (Alertmanager) • unified exploration (Grafana)</text>
+
+  <rect x="45" y="150" width="300" height="750" class="stage"/><text x="70" y="186" class="layer">WORKLOADS &amp; HOSTS</text>
+  <rect x="370" y="150" width="300" height="750" class="stage"/><text x="395" y="186" class="layer">COLLECT &amp; EXPORT</text>
+  <rect x="695" y="150" width="390" height="750" class="stage"/><text x="720" y="186" class="layer">STORE &amp; QUERY</text>
+  <rect x="1110" y="150" width="295" height="750" class="stage"/><text x="1135" y="186" class="layer">ALERT PIPELINE</text>
+  <rect x="1430" y="150" width="445" height="750" class="stage"/><text x="1455" y="186" class="layer">EXPLORE &amp; CORRELATE</text>
+
+  <g><rect x="70" y="230" width="250" height="140" class="box"/><circle cx="110" cy="273" r="22" fill="#55d187"/><text x="102" y="281" font-size="22" font-weight="850">M</text><text x="148" y="275" class="boxtitle">App metrics</text><text x="95" y="318" class="meta">/metrics endpoint</text><text x="95" y="343" class="meta">Prometheus exposition</text></g>
+  <g><rect x="70" y="425" width="250" height="140" class="box"/><circle cx="110" cy="468" r="22" fill="#47b7ff"/><text x="103" y="476" font-size="22" font-weight="850">L</text><text x="148" y="470" class="boxtitle">Container logs</text><text x="95" y="513" class="meta">Docker socket / files</text><text x="95" y="538" class="meta">structured log records</text></g>
+  <g><rect x="70" y="620" width="250" height="140" class="box"/><circle cx="110" cy="663" r="22" fill="#a987ff"/><text x="103" y="671" font-size="22" font-weight="850">T</text><text x="148" y="665" class="boxtitle">Instrumented apps</text><text x="95" y="708" class="meta">OpenTelemetry SDK</text><text x="95" y="733" class="meta">spans + resource attrs</text></g>
+
+  <g><rect x="400" y="270" width="240" height="290" class="box" stroke="#ff9a3d"/><circle cx="520" cy="342" r="42" fill="#a95014"/><text x="501" y="355" font-size="34" font-weight="850">A</text><text x="520" y="415" class="boxtitle" text-anchor="middle">Grafana Alloy</text><text x="520" y="450" class="meta" text-anchor="middle">discover • label • batch</text><rect x="435" y="478" width="170" height="34" rx="17" class="pill"/><text x="520" y="500" class="pilltext" text-anchor="middle">OTLP 4317 / 4318</text><rect x="455" y="520" width="130" height="27" rx="14" class="pill"/><text x="520" y="539" class="pilltext" text-anchor="middle">UI 12345</text></g>
+  <g><rect x="400" y="635" width="240" height="150" class="box" stroke="#55d187"/><text x="520" y="685" class="boxtitle" text-anchor="middle">Node Exporter</text><text x="520" y="722" class="meta" text-anchor="middle">host CPU / RAM / disk</text><rect x="460" y="742" width="120" height="28" rx="14" class="pill"/><text x="520" y="762" class="pilltext" text-anchor="middle">TCP 9100</text></g>
+
+  <g><rect x="730" y="220" width="320" height="170" class="box" stroke="#55d187"/><text x="890" y="270" class="boxtitle" text-anchor="middle">Prometheus</text><text x="890" y="308" class="meta" text-anchor="middle">scrape • TSDB • PromQL • rules</text><rect x="812" y="335" width="156" height="30" rx="15" class="pill"/><text x="890" y="356" class="pilltext" text-anchor="middle">HTTP 9090</text></g>
+  <g><rect x="730" y="455" width="320" height="170" class="box" stroke="#47b7ff"/><text x="890" y="505" class="boxtitle" text-anchor="middle">Loki</text><text x="890" y="543" class="meta" text-anchor="middle">labels • chunks • LogQL</text><rect x="812" y="570" width="156" height="30" rx="15" class="pill"/><text x="890" y="591" class="pilltext" text-anchor="middle">HTTP 3100</text></g>
+  <g><rect x="730" y="690" width="320" height="170" class="box" stroke="#a987ff"/><text x="890" y="740" class="boxtitle" text-anchor="middle">Tempo</text><text x="890" y="778" class="meta" text-anchor="middle">trace blocks • TraceQL</text><rect x="785" y="805" width="210" height="30" rx="15" class="pill"/><text x="890" y="826" class="pilltext" text-anchor="middle">query 3200 • OTLP 4317</text></g>
+
+  <g><rect x="1145" y="300" width="225" height="200" class="box" stroke="#ff5f6d"/><text x="1257" y="355" class="boxtitle" text-anchor="middle">Alertmanager</text><text x="1257" y="393" class="meta" text-anchor="middle">group • dedupe • route</text><rect x="1182" y="420" width="150" height="30" rx="15" class="pill"/><text x="1257" y="441" class="pilltext" text-anchor="middle">HTTP 9093</text></g>
+  <g><rect x="1145" y="610" width="225" height="150" class="box" stroke="#ff5f6d"/><text x="1257" y="660" class="boxtitle" text-anchor="middle">Receivers</text><text x="1257" y="700" class="meta" text-anchor="middle">Slack • Email • PagerDuty</text><text x="1257" y="730" class="meta" text-anchor="middle">webhook / SMTP</text></g>
+
+  <g><rect x="1470" y="300" width="365" height="350" class="box" stroke="#ff9a3d"/><circle cx="1652" cy="388" r="50" fill="#ba5a14"/><text x="1631" y="404" font-size="42" font-weight="850">G</text><text x="1652" y="477" class="boxtitle" text-anchor="middle">Grafana</text><text x="1652" y="518" class="meta" text-anchor="middle">dashboards • Explore • correlations</text><rect x="1547" y="548" width="210" height="34" rx="17" class="pill"/><text x="1652" y="570" class="pilltext" text-anchor="middle">Metrics ↔ Logs ↔ Traces</text><rect x="1592" y="596" width="120" height="30" rx="15" class="pill"/><text x="1652" y="617" class="pilltext" text-anchor="middle">TCP 3000</text></g>
+
+  <path d="M320 300 C360 300 360 315 400 315" class="metric"/><text x="354" y="286" class="label" text-anchor="middle">scrape</text>
+  <path d="M320 495 H400" class="log"/><text x="360" y="480" class="label" text-anchor="middle">tail</text>
+  <path d="M320 690 C360 690 360 515 400 515" class="trace"/><text x="355" y="655" class="label" text-anchor="middle">OTLP</text>
+  <path d="M640 350 C680 350 690 305 730 305" class="metric"/><text x="684" y="292" class="label" text-anchor="middle">/metrics</text>
+  <path d="M640 485 C680 485 690 540 730 540" class="log"/><text x="685" y="513" class="label" text-anchor="middle">push API</text>
+  <path d="M640 520 C690 520 680 775 730 775" class="trace"/><text x="681" y="728" class="label" text-anchor="middle">OTLP/gRPC</text>
+  <path d="M640 710 C690 710 680 340 730 340" class="metric"/><text x="670" y="630" class="label" text-anchor="middle">scrape :9100</text>
+  <path d="M1050 305 H1145" class="alert"/><text x="1097" y="286" class="label" text-anchor="middle">firing alerts</text>
+  <path d="M1257 500 V610" class="alert"/><text x="1305" y="560" class="label">route</text>
+  <path d="M1050 305 C1210 210 1370 260 1470 365" class="query"/><path d="M1050 540 C1210 540 1340 470 1470 470" class="query"/><path d="M1050 775 C1220 775 1350 625 1470 575" class="query"/><text x="1300" y="452" class="label" text-anchor="middle">PromQL / LogQL / TraceQL</text>
+  <path d="M1050 760 C1090 720 1090 390 1050 350" class="metric"/><text x="1092" y="560" class="label" text-anchor="middle" transform="rotate(-90 1092 560)">span metrics + exemplars → remote_write</text>
+
+  <rect x="70" y="945" width="1780" height="80" rx="18" fill="#0c1728" stroke="#263b58"/>
+  <line x1="105" y1="982" x2="170" y2="982" class="metric"/><text x="190" y="988" class="meta">metrics</text><line x1="340" y1="982" x2="405" y2="982" class="log"/><text x="425" y="988" class="meta">logs</text><line x1="555" y1="982" x2="620" y2="982" class="trace"/><text x="640" y="988" class="meta">traces</text><line x1="780" y1="982" x2="845" y2="982" class="query"/><text x="865" y="988" class="meta">query + response</text><line x1="1080" y1="982" x2="1145" y2="982" class="alert"/><text x="1165" y="988" class="meta">alerts</text><text x="1390" y="988" class="meta">Correlation key: trace_id / exemplars</text>
+</svg>
+
@@ -0,0 +1,29 @@
+---
+# Deployment base path on the target host.
+observability_root: /opt/observability
+
+# Change these tags deliberately when upgrading.
+elastic_version: "9.1.3"
+grafana_version: "12.1.1"
+prometheus_version: "v3.5.0"
+alertmanager_version: "v0.28.1"
+loki_version: "3.5.3"
+tempo_version: "2.8.2"
+alloy_version: "v1.10.2"
+node_exporter_version: "v1.9.1"
+
+# Lab-safe defaults. Override with Ansible Vault for real deployments.
+grafana_admin_user: admin
+grafana_admin_password: "change-me-now"
+
+# Optional Alertmanager webhook. Leave empty to install a no-op receiver.
+alertmanager_webhook_url: ""
+
+# Add application /metrics endpoints here. Example:
+# prometheus_extra_scrape_jobs:
+#   - job_name: checkout-api
+#     targets: ["10.20.0.15:8080", "10.20.0.16:8080"]
+prometheus_extra_scrape_jobs: []
+
+# JVM heap for single-node Elasticsearch. Keep Xms and Xmx equal.
+elasticsearch_heap: 1g
@@ -0,0 +1,6 @@
+---
+all:
+  children:
+    observability:
+      hosts:
+        cloudzy:
@@ -0,0 +1,10 @@
+---
+- name: Deploy ELK and Grafana
+  hosts: observability
+  become: true
+  vars:
+    observability_stack: elk-grafana
+  roles:
+    - docker
+    - observability
+
@@ -0,0 +1,10 @@
+---
+- name: Deploy Grafana observability stack
+  hosts: observability
+  become: true
+  vars:
+    observability_stack: grafana-stack
+  roles:
+    - docker
+    - observability
+
@@ -0,0 +1,7 @@
+---
+collections:
+  - name: community.docker
+    version: ">=4.0.0"
+  - name: ansible.posix
+    version: ">=2.0.0"
+
@@ -0,0 +1,56 @@
+---
+- name: Assert supported operating system
+  ansible.builtin.assert:
+    that:
+      - ansible_facts.os_family == 'Debian'
+      - ansible_facts.distribution in ['Ubuntu', 'Debian']
+    fail_msg: This role supports Ubuntu and Debian hosts.
+
+- name: Install Docker repository prerequisites
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - curl
+      - gnupg
+    state: present
+    update_cache: true
+
+- name: Create APT keyrings directory
+  ansible.builtin.file:
+    path: /etc/apt/keyrings
+    state: directory
+    mode: "0755"
+
+- name: Download Docker signing key
+  ansible.builtin.get_url:
+    url: "https://download.docker.com/linux/{{ ansible_facts.distribution | lower }}/gpg"
+    dest: /etc/apt/keyrings/docker.asc
+    mode: "0644"
+
+- name: Configure Docker APT repository
+  ansible.builtin.apt_repository:
+    repo: >-
+      deb [arch={{ {'x86_64':'amd64','aarch64':'arm64'}.get(ansible_facts.architecture, ansible_facts.architecture) }}
+      signed-by=/etc/apt/keyrings/docker.asc]
+      https://download.docker.com/linux/{{ ansible_facts.distribution | lower }}
+      {{ ansible_facts.distribution_release }} stable
+    filename: docker
+    state: present
+
+- name: Install Docker Engine and Compose plugin
+  ansible.builtin.apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-buildx-plugin
+      - docker-compose-plugin
+    state: present
+    update_cache: true
+
+- name: Enable and start Docker
+  ansible.builtin.systemd_service:
+    name: docker
+    state: started
+    enabled: true
+
@@ -0,0 +1,7 @@
+---
+- name: Restart observability stack
+  community.docker.docker_compose_v2:
+    project_src: "{{ stack_root }}"
+    state: restarted
+  when: stack_root is defined
+
@@ -0,0 +1,90 @@
+---
+- name: Validate selected stack
+  ansible.builtin.assert:
+    that:
+      - observability_stack in ['elk-grafana', 'grafana-stack']
+    fail_msg: observability_stack must be elk-grafana or grafana-stack.
+
+- name: Set stack deployment directory
+  ansible.builtin.set_fact:
+    stack_root: "{{ observability_root }}/{{ observability_stack }}"
+
+- name: Create stack directory tree
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  loop: >-
+    {{
+      [stack_root, stack_root + '/config']
+      + (
+          [stack_root + '/config/filebeat', stack_root + '/config/logstash/pipeline', stack_root + '/config/grafana/provisioning/datasources']
+          if observability_stack == 'elk-grafana'
+          else [stack_root + '/config/prometheus/rules', stack_root + '/config/alertmanager', stack_root + '/config/loki', stack_root + '/config/tempo', stack_root + '/config/alloy', stack_root + '/config/grafana/provisioning/datasources']
+        )
+    }}
+
+- name: Apply Elasticsearch kernel requirement
+  ansible.posix.sysctl:
+    name: vm.max_map_count
+    value: "262144"
+    state: present
+    sysctl_set: true
+    reload: true
+  when: observability_stack == 'elk-grafana'
+
+- name: Render ELK and Grafana files
+  ansible.builtin.template:
+    src: "elk-grafana/{{ item.src }}"
+    dest: "{{ stack_root }}/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode | default('0644') }}"
+  loop:
+    - { src: docker-compose.yml.j2, dest: docker-compose.yml }
+    - { src: config/filebeat/filebeat.yml.j2, dest: config/filebeat/filebeat.yml }
+    - { src: config/logstash/pipeline/logstash.conf.j2, dest: config/logstash/pipeline/logstash.conf }
+    - { src: config/grafana/provisioning/datasources/datasources.yml.j2, dest: config/grafana/provisioning/datasources/datasources.yml }
+  when: observability_stack == 'elk-grafana'
+  notify: Restart observability stack
+
+- name: Render Grafana stack files
+  ansible.builtin.template:
+    src: "grafana-stack/{{ item.src }}"
+    dest: "{{ stack_root }}/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode | default('0644') }}"
+  loop:
+    - { src: docker-compose.yml.j2, dest: docker-compose.yml }
+    - { src: config/prometheus/prometheus.yml.j2, dest: config/prometheus/prometheus.yml }
+    - { src: config/prometheus/rules/host.rules.yml.j2, dest: config/prometheus/rules/host.rules.yml }
+    - { src: config/alertmanager/alertmanager.yml.j2, dest: config/alertmanager/alertmanager.yml }
+    - { src: config/loki/loki.yml.j2, dest: config/loki/loki.yml }
+    - { src: config/tempo/tempo.yml.j2, dest: config/tempo/tempo.yml }
+    - { src: config/alloy/config.alloy.j2, dest: config/alloy/config.alloy }
+    - { src: config/grafana/provisioning/datasources/datasources.yml.j2, dest: config/grafana/provisioning/datasources/datasources.yml }
+  when: observability_stack == 'grafana-stack'
+  notify: Restart observability stack
+
+- name: Start selected observability stack
+  community.docker.docker_compose_v2:
+    project_src: "{{ stack_root }}"
+    state: present
+    pull: missing
+    remove_orphans: true
+
+- name: Flush restart handler before health checks
+  ansible.builtin.meta: flush_handlers
+
+- name: Wait for Grafana
+  ansible.builtin.uri:
+    url: http://127.0.0.1:3000/api/health
+    status_code: 200
+  register: grafana_health
+  retries: 30
+  delay: 5
+  until: grafana_health.status == 200
+
@@ -0,0 +1,21 @@
+filebeat.inputs:
+  - type: filestream
+    id: docker-containers
+    enabled: true
+    paths:
+      - /var/lib/docker/containers/*/*.log
+    parsers:
+      - container:
+          stream: all
+          format: docker
+
+processors:
+  - add_docker_metadata:
+      host: unix:///var/run/docker.sock
+  - add_host_metadata: ~
+
+output.logstash:
+  hosts: ["logstash:5044"]
+
+logging.level: info
+
@@ -0,0 +1,18 @@
+apiVersion: 1
+
+datasources:
+  - name: Elasticsearch
+    uid: elasticsearch
+    type: elasticsearch
+    access: proxy
+    url: http://elasticsearch:9200
+    database: "logs-*"
+    isDefault: true
+    editable: true
+    jsonData:
+      index: "logs-*"
+      interval: Daily
+      timeField: "@timestamp"
+      logMessageField: message
+      logLevelField: log.level
+
@@ -0,0 +1,30 @@
+input {
+  beats {
+    port => 5044
+  }
+  tcp {
+    port => 5000
+    codec => json_lines
+  }
+}
+
+filter {
+  if [message] =~ /^\s*\{/ {
+    json {
+      source => "message"
+      skip_on_invalid_json => true
+    }
+  }
+  mutate {
+    add_field => { "[@metadata][target_index]" => "logs-%{+YYYY.MM.dd}" }
+  }
+}
+
+output {
+  elasticsearch {
+    hosts => ["http://elasticsearch:9200"]
+    index => "%{[@metadata][target_index]}"
+  }
+  stdout { codec => dots }
+}
+
@@ -0,0 +1,86 @@
+services:
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:{{ elastic_version }}
+    restart: unless-stopped
+    environment:
+      discovery.type: single-node
+      xpack.security.enabled: "false"
+      xpack.security.enrollment.enabled: "false"
+      ES_JAVA_OPTS: "-Xms{{ elasticsearch_heap }} -Xmx{{ elasticsearch_heap }}"
+    volumes:
+      - elasticsearch-data:/usr/share/elasticsearch/data
+    ports:
+      - "9200:9200"
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://localhost:9200/_cluster/health || exit 1"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+    networks: [observability]
+
+  logstash:
+    image: docker.elastic.co/logstash/logstash:{{ elastic_version }}
+    restart: unless-stopped
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+    volumes:
+      - ./config/logstash/pipeline:/usr/share/logstash/pipeline:ro
+      - logstash-data:/usr/share/logstash/data
+    ports:
+      - "5044:5044"
+      - "5000:5000/tcp"
+    networks: [observability]
+
+  filebeat:
+    image: docker.elastic.co/beats/filebeat:{{ elastic_version }}
+    user: root
+    restart: unless-stopped
+    command: ["filebeat", "-e", "--strict.perms=false"]
+    depends_on: [logstash]
+    volumes:
+      - ./config/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - filebeat-data:/usr/share/filebeat/data
+    networks: [observability]
+
+  kibana:
+    image: docker.elastic.co/kibana/kibana:{{ elastic_version }}
+    restart: unless-stopped
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+    environment:
+      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
+    ports:
+      - "5601:5601"
+    networks: [observability]
+
+  grafana:
+    image: grafana/grafana:{{ grafana_version }}
+    restart: unless-stopped
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+    environment:
+      GF_SECURITY_ADMIN_USER: "{{ grafana_admin_user }}"
+      GF_SECURITY_ADMIN_PASSWORD: "{{ grafana_admin_password }}"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
+    ports:
+      - "3000:3000"
+    networks: [observability]
+
+volumes:
+  elasticsearch-data:
+  logstash-data:
+  filebeat-data:
+  grafana-data:
+
+networks:
+  observability:
+    driver: bridge
+
@@ -0,0 +1,18 @@
+global:
+  resolve_timeout: 5m
+
+route:
+  receiver: default
+  group_by: [alertname, instance]
+  group_wait: 30s
+  group_interval: 5m
+  repeat_interval: 4h
+
+receivers:
+  - name: default
+{% if alertmanager_webhook_url | length > 0 %}
+    webhook_configs:
+      - url: "{{ alertmanager_webhook_url }}"
+        send_resolved: true
+{% endif %}
+
@@ -0,0 +1,51 @@
+logging {
+  level = "info"
+}
+
+discovery.docker "local" {
+  host = "unix:///var/run/docker.sock"
+}
+
+discovery.relabel "docker" {
+  targets = discovery.docker.local.targets
+
+  rule {
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/(.*)"
+    target_label  = "container"
+  }
+}
+
+loki.source.docker "local" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.docker.output
+  forward_to = [loki.write.default.receiver]
+}
+
+loki.write "default" {
+  endpoint {
+    url = "http://loki:3100/loki/api/v1/push"
+  }
+}
+
+otelcol.receiver.otlp "default" {
+  grpc {
+    endpoint = "0.0.0.0:4317"
+  }
+  http {
+    endpoint = "0.0.0.0:4318"
+  }
+  output {
+    traces = [otelcol.exporter.otlp.tempo.input]
+  }
+}
+
+otelcol.exporter.otlp "tempo" {
+  client {
+    endpoint = "tempo:4317"
+    tls {
+      insecure = true
+    }
+  }
+}
+
@@ -0,0 +1,54 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    uid: prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: true
+    jsonData:
+      exemplarTraceIdDestinations:
+        - datasourceUid: tempo
+          name: trace_id
+
+  - name: Loki
+    uid: loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    editable: true
+    jsonData:
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
+          name: TraceID
+          url: '$${__value.raw}'
+
+  - name: Tempo
+    uid: tempo
+    type: tempo
+    access: proxy
+    url: http://tempo:3200
+    editable: true
+    jsonData:
+      tracesToLogsV2:
+        datasourceUid: loki
+        filterByTraceID: true
+        spanStartTimeShift: -1m
+        spanEndTimeShift: 1m
+      serviceMap:
+        datasourceUid: prometheus
+      nodeGraph:
+        enabled: true
+
+  - name: Alertmanager
+    uid: alertmanager
+    type: alertmanager
+    access: proxy
+    url: http://alertmanager:9093
+    editable: true
+    jsonData:
+      implementation: prometheus
+
@@ -0,0 +1,34 @@
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+
+common:
+  path_prefix: /var/loki
+  replication_factor: 1
+  ring:
+    instance_addr: 127.0.0.1
+    kvstore:
+      store: inmemory
+
+schema_config:
+  configs:
+    - from: 2024-01-01
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+storage_config:
+  filesystem:
+    directory: /var/loki/chunks
+
+compactor:
+  working_directory: /var/loki/compactor
+
+limits_config:
+  retention_period: 336h
+  allow_structured_metadata: true
+
@@ -0,0 +1,43 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ["alertmanager:9093"]
+
+scrape_configs:
+  - job_name: prometheus
+    static_configs:
+      - targets: ["prometheus:9090"]
+
+  - job_name: node-exporter
+    static_configs:
+      - targets: ["node-exporter:9100"]
+
+  - job_name: loki
+    static_configs:
+      - targets: ["loki:3100"]
+
+  - job_name: tempo
+    static_configs:
+      - targets: ["tempo:3200"]
+
+  - job_name: alloy
+    static_configs:
+      - targets: ["alloy:12345"]
+
+  - job_name: grafana
+    static_configs:
+      - targets: ["grafana:3000"]
+
+{% for job in prometheus_extra_scrape_jobs %}
+  - job_name: "{{ job.job_name }}"
+    static_configs:
+      - targets: {{ job.targets | to_json }}
+{% endfor %}
+
@@ -0,0 +1,21 @@
+groups:
+  - name: host.rules
+    rules:
+      - alert: HostExporterDown
+        expr: up{job="node-exporter"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Node exporter is unavailable"
+          description: "Prometheus has failed to scrape {{ '{{' }} $labels.instance {{ '}}' }} for more than two minutes."
+
+      - alert: HostHighCPU
+        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High CPU utilization"
+          description: "CPU utilization on {{ '{{' }} $labels.instance {{ '}}' }} has exceeded 85% for ten minutes."
+
@@ -0,0 +1,42 @@
+server:
+  http_listen_port: 3200
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: 0.0.0.0:4317
+        http:
+          endpoint: 0.0.0.0:4318
+
+ingester:
+  max_block_duration: 5m
+
+compactor:
+  compaction:
+    block_retention: 336h
+
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /var/tempo/wal
+    local:
+      path: /var/tempo/blocks
+
+metrics_generator:
+  registry:
+    external_labels:
+      source: tempo
+  storage:
+    path: /var/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+
+overrides:
+  defaults:
+    metrics_generator:
+      processors: [service-graphs, span-metrics, local-blocks]
+
@@ -0,0 +1,110 @@
+services:
+  prometheus:
+    image: prom/prometheus:{{ prometheus_version }}
+    restart: unless-stopped
+    command:
+      - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.path=/prometheus
+      - --storage.tsdb.retention.time=15d
+      - --web.enable-lifecycle
+      - --web.enable-remote-write-receiver
+    volumes:
+      - ./config/prometheus:/etc/prometheus:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    networks: [observability]
+
+  alertmanager:
+    image: prom/alertmanager:{{ alertmanager_version }}
+    restart: unless-stopped
+    command:
+      - --config.file=/etc/alertmanager/alertmanager.yml
+      - --storage.path=/alertmanager
+    volumes:
+      - ./config/alertmanager:/etc/alertmanager:ro
+      - alertmanager-data:/alertmanager
+    ports:
+      - "9093:9093"
+    networks: [observability]
+
+  loki:
+    image: grafana/loki:{{ loki_version }}
+    restart: unless-stopped
+    command: ["-config.file=/etc/loki/loki.yml"]
+    volumes:
+      - ./config/loki/loki.yml:/etc/loki/loki.yml:ro
+      - loki-data:/var/loki
+    ports:
+      - "3100:3100"
+    networks: [observability]
+
+  tempo:
+    image: grafana/tempo:{{ tempo_version }}
+    restart: unless-stopped
+    command: ["-config.file=/etc/tempo/tempo.yml"]
+    volumes:
+      - ./config/tempo/tempo.yml:/etc/tempo/tempo.yml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200"
+    networks: [observability]
+
+  node-exporter:
+    image: prom/node-exporter:{{ node_exporter_version }}
+    restart: unless-stopped
+    command:
+      - --path.rootfs=/host
+    pid: host
+    volumes:
+      - /:/host:ro,rslave
+    ports:
+      - "9100:9100"
+    networks: [observability]
+
+  alloy:
+    image: grafana/alloy:{{ alloy_version }}
+    restart: unless-stopped
+    command:
+      - run
+      - --server.http.listen-addr=0.0.0.0:12345
+      - --storage.path=/var/lib/alloy/data
+      - /etc/alloy/config.alloy
+    volumes:
+      - ./config/alloy/config.alloy:/etc/alloy/config.alloy:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - alloy-data:/var/lib/alloy/data
+    ports:
+      - "12345:12345"
+      - "4317:4317"
+      - "4318:4318"
+    networks: [observability]
+
+  grafana:
+    image: grafana/grafana:{{ grafana_version }}
+    restart: unless-stopped
+    depends_on: [prometheus, loki, tempo]
+    environment:
+      GF_SECURITY_ADMIN_USER: "{{ grafana_admin_user }}"
+      GF_SECURITY_ADMIN_PASSWORD: "{{ grafana_admin_password }}"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+      GF_FEATURE_TOGGLES_ENABLE: traceqlEditor
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
+    ports:
+      - "3000:3000"
+    networks: [observability]
+
+volumes:
+  prometheus-data:
+  alertmanager-data:
+  loki-data:
+  tempo-data:
+  alloy-data:
+  grafana-data:
+
+networks:
+  observability:
+    driver: bridge
+