Add observability platform Ansible project

Deploys the Grafana stack (Prometheus/Alertmanager/Loki/Tempo/Grafana with Alloy as collector) and the ELK stack (Filebeat/Logstash/Elasticsearch/ Kibana) via Docker Compose on a target host. External to the application repo, pre-built by the mentor.
2026-09-05 08:11:34 +03:30
parent eb99cdbb8c
commit bffc42c9d3
26 changed files with 1056 additions and 0 deletions
@@ -0,0 +1,204 @@
 # Deploying Two Observability Stacks with Ansible
 This project deploys two independent stacks on a single Ubuntu/Debian host using Docker Compose:
 1. **ELK + Grafana:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana
 2. **Grafana Observability:** Prometheus + Alertmanager + Loki + Tempo + Grafana, along with Alloy and Node Exporter for data collection
 > The default settings are suitable for Lab, Demo, and training purposes. For Production, be sure to read the "Production Notes" section.
 ## Operational Diagrams
 - `diagrams/01-elk-grafana-operational.png` and its SVG vector version
 - `diagrams/02-grafana-lgtm-operational.png` and its SVG vector version
 Path colors in the second diagram:
 - Green: Metrics
 - Blue: Logs
 - Purple: Traces
 - Dashed orange: Query/Response
 - Red: Alert delivery
 ## Project Structure
 ```text
 inventory/hosts.yml                 SSH targets
 group_vars/all.yml                  Versions, Grafana password, and variables
 playbooks/elk-grafana.yml           Installs ELK + Grafana
 playbooks/grafana-stack.yml         Installs Grafana/Prometheus/Loki/Tempo
 roles/docker/                       Installs Docker Engine and Compose v2
 roles/observability/                Templates, execution, and health checks
 diagrams/                           PNG and SVG diagrams
 ```
 ## Prerequisites
 - Control node: Ansible Core 2.16 or newer
 - Target: Ubuntu 22.04/24.04 or Debian 12, with SSH and sudo access
 - Recommended minimum for the ELK Lab: 4 vCPUs, 8 GB RAM, and 50 GB disk
 - Recommended minimum for the Grafana Stack Lab: 2 vCPUs, 4 GB RAM, and 30 GB disk
 On the control node:
 ```bash
 cd observability-ansible
 ansible-galaxy collection install -r requirements.yml
 ```
 ## 1. Configure the Inventory
 Edit the `inventory/hosts.yml` file:
 ```yaml
 all:
  children:
    observability:
      hosts:
        obs01:
          ansible_host: 10.10.20.15
          ansible_user: ubuntu
          ansible_ssh_private_key_file: ~/.ssh/id_ed25519
 ```
 Test the connection:
 ```bash
 ansible observability -m ping
 ```
 ## 2. Configure Variables and Secrets
 Versions and core settings live in `group_vars/all.yml`. Be sure to change the default Grafana password. To store the password securely:
 ```bash
 ansible-vault encrypt_string 'A-strong-password' --name grafana_admin_password
 ```
 Replace the `grafana_admin_password` value with the Vault output, and run the playbook with `--ask-vault-pass`.
 If your application exposes a `/metrics` endpoint, add it here:
 ```yaml
 prometheus_extra_scrape_jobs:
  - job_name: checkout-api
    targets:
      - 10.20.0.15:8080
      - 10.20.0.16:8080
 ```
 To send alerts to a webhook, also set the following value:
 ```yaml
 alertmanager_webhook_url: "https://example.internal/alerts"
 ```
 ## 3. Install ELK + Grafana
 ```bash
 ansible-playbook playbooks/elk-grafana.yml
 ```
 Endpoints:
 | Service | URL/Port | Purpose |
 | --- | --- | --- |
 | Elasticsearch | `http://SERVER:9200` | Index and Search API |
 | Logstash Beats input | `SERVER:5044` | Receives Filebeat |
 | Logstash JSON/TCP | `SERVER:5000` | Testing and direct JSON Lines ingest |
 | Kibana | `http://SERVER:5601` | Discover and Elastic analysis |
 | Grafana | `http://SERVER:3000` | Dashboards on top of Elasticsearch |
 Filebeat in this project reads the Docker container logs on the same host and sends them to Logstash. Logstash parses the JSON and writes it into the daily `logs-YYYY.MM.dd` index.
 Test ingest:
 ```bash
 printf '%s\n' '{"message":"checkout completed","level":"info","service":"checkout"}' \
  | nc SERVER 5000
 curl 'http://SERVER:9200/logs-*/_search?pretty'
 ```
 ## 4. Install Grafana + Prometheus + Alertmanager + Loki + Tempo
 ```bash
 ansible-playbook playbooks/grafana-stack.yml
 ```
 Endpoints:
 | Service | URL/Port | Purpose |
 | --- | --- | --- |
 | Grafana | `http://SERVER:3000` | Explore, Dashboards, and correlation |
 | Prometheus | `http://SERVER:9090` | Metrics, PromQL, and alert rules |
 | Alertmanager | `http://SERVER:9093` | Grouping, deduplication, and routing of alerts |
 | Loki | `http://SERVER:3100` | Log ingestion/query and LogQL |
 | Tempo | `http://SERVER:3200` | Trace query and TraceQL |
 | Alloy UI | `http://SERVER:12345` | Collector pipeline status |
 | OTLP/gRPC | `SERVER:4317` | Receives traces from the application |
 | OTLP/HTTP | `SERVER:4318` | Receives traces from the application |
 | Node Exporter | `http://SERVER:9100/metrics` | Host-level metrics |
 The Prometheus, Loki, Tempo, and Alertmanager data sources are automatically provisioned in Grafana. Grafana uses exemplars for Metrics → Trace, `trace_id` for Logs → Trace, and Prometheus for the Tempo service graph.
 Example OpenTelemetry environment variables for your application:
 ```bash
 export OTEL_SERVICE_NAME=checkout-api
 export OTEL_EXPORTER_OTLP_ENDPOINT=http://SERVER:4318
 export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
 ```
 Alloy pushes Docker container logs to Loki and forwards OTLP traces to Tempo. Prometheus scrapes internal services and Node Exporter. The sample rules `HostExporterDown` and `HostHighCPU` are also installed.
 ## Day-to-Day Operations
 On the target server:
 ```bash
 cd /opt/observability/elk-grafana
 docker compose ps
 docker compose logs -f --tail=100
 cd /opt/observability/grafana-stack
 docker compose ps
 docker compose logs -f --tail=100
 ```
 Re-running the playbook is idempotent; changing a template or version tag triggers a recreate/restart of the relevant stack. Data is kept in Docker named volumes.
 ## Removing a Stack
 To stop and remove the containers without deleting the data:
 ```bash
 cd /opt/observability/STACK_NAME
 docker compose down
 ```
 The `-v` flag deletes all volumes and monitoring data; only use it when you have a backup and specifically intend to do this.
 ## Production Notes
 The default playbook intentionally builds a single-node, easy-to-understand deployment. For Production:
 - Enable TLS, authentication, and RBAC on all endpoints; don't expose storage ports directly to the internet.
 - Keep secrets in Ansible Vault or a Secret Manager.
 - Run Elasticsearch with at least three nodes, a proper snapshot repository, and ILM.
 - Use S3-compatible object storage and replication for Loki and Tempo.
 - Design Prometheus with remote-write/Thanos/Mimir, retention, and adequate disk capacity.
 - Run Alertmanager in HA and test your notification routes.
 - Add a reverse proxy, DNS, certificates, firewall rules, and backup/restore testing.
 - Upgrade versions in staging first; they are intentionally pinned in `group_vars/all.yml`.
 ## Official Resources
 - [Elastic Docker deployment](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-docker-compose)
 - [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
 - [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
 - [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/)
 - [Grafana Loki configuration](https://grafana.com/docs/loki/latest/configure/)
 - [Grafana Tempo configuration](https://grafana.com/docs/tempo/latest/configuration/)
 - [Grafana Alloy](https://grafana.com/docs/alloy/latest/)
 - [Ansible Docker Compose v2 module](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_module.html)
@@ -0,0 +1,10 @@
 [defaults]
 inventory = inventory/hosts.yml
 roles_path = roles
 host_key_checking = False
 interpreter_python = auto_silent
 retry_files_enabled = False
 [ssh_connection]
 pipelining = True
@@ -0,0 +1,44 @@
 <svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080" viewBox="0 0 1920 1080">
  <defs>
    <linearGradient id="bg" x1="0" y1="0" x2="1" y2="1"><stop stop-color="#07111f"/><stop offset="1" stop-color="#111a2d"/></linearGradient>
    <filter id="shadow"><feDropShadow dx="0" dy="10" stdDeviation="12" flood-opacity=".25"/></filter>
    <marker id="blueArrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#35b9ff"/></marker>
    <marker id="orangeArrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#ff9f43"/></marker>
    <marker id="redArrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#ff5f6d"/></marker>
    <style>
      text{font-family:Inter,Segoe UI,Arial,sans-serif;fill:#eaf2ff}.title{font-size:42px;font-weight:800}.sub{font-size:18px;fill:#9fb0c9}.layer{font-size:15px;font-weight:700;letter-spacing:2px;fill:#7185a3}.box{fill:#121f35;stroke:#314762;stroke-width:2;rx:18;filter:url(#shadow)}.boxtitle{font-size:23px;font-weight:750}.meta{font-size:15px;fill:#afbdd1}.pill{fill:#1d304d;stroke:#405a7a}.pilltext{font-size:14px;fill:#d7e6fb}.flow{fill:none;stroke:#35b9ff;stroke-width:4;marker-end:url(#blueArrow)}.query{fill:none;stroke:#ff9f43;stroke-width:4;stroke-dasharray:12 9;marker-start:url(#orangeArrow);marker-end:url(#orangeArrow)}.alert{fill:none;stroke:#ff5f6d;stroke-width:4;marker-end:url(#redArrow)}.label{font-size:14px;font-weight:650;paint-order:stroke;stroke:#0a1424;stroke-width:7px;stroke-linejoin:round}.stage{fill:#0b1728;stroke:#223651;stroke-width:1.5;rx:22}
    </style>
  </defs>
  <rect width="1920" height="1080" fill="url(#bg)"/>
  <text x="70" y="74" class="title">ELK + Grafana — Operational Data Flow</text>
  <text x="70" y="108" class="sub">Collection → processing → indexed storage → search, dashboards and alert delivery</text>
  <rect x="45" y="150" width="295" height="750" class="stage"/><text x="70" y="186" class="layer">DATA SOURCES</text>
  <rect x="365" y="150" width="285" height="750" class="stage"/><text x="390" y="186" class="layer">COLLECTION</text>
  <rect x="675" y="150" width="285" height="750" class="stage"/><text x="700" y="186" class="layer">PROCESSING</text>
  <rect x="985" y="150" width="390" height="750" class="stage"/><text x="1010" y="186" class="layer">INDEXED STORAGE</text>
  <rect x="1400" y="150" width="475" height="750" class="stage"/><text x="1425" y="186" class="layer">QUERY, VISUALIZE &amp; ALERT</text>
  <g><rect x="70" y="235" width="245" height="130" class="box"/><circle cx="108" cy="277" r="22" fill="#4ea1ff"/><text x="101" y="285" font-size="24" font-weight="800">A</text><text x="145" y="278" class="boxtitle">Applications</text><text x="95" y="320" class="meta">JSON / text logs</text><text x="95" y="344" class="meta">trace_id in log context</text></g>
  <g><rect x="70" y="430" width="245" height="130" class="box"/><circle cx="108" cy="472" r="22" fill="#6ad36a"/><text x="101" y="480" font-size="23" font-weight="800">S</text><text x="145" y="473" class="boxtitle">System logs</text><text x="95" y="515" class="meta">journald / syslog</text><text x="95" y="539" class="meta">auth and kernel events</text></g>
  <g><rect x="70" y="625" width="245" height="130" class="box"/><circle cx="108" cy="667" r="22" fill="#9c7cff"/><text x="101" y="675" font-size="23" font-weight="800">C</text><text x="145" y="668" class="boxtitle">Containers</text><text x="95" y="710" class="meta">Docker JSON logs</text><text x="95" y="734" class="meta">Kubernetes log files</text></g>
  <g><rect x="395" y="340" width="225" height="250" class="box" stroke="#36c5f0"/><circle cx="507" cy="405" r="38" fill="#16769a"/><text x="493" y="417" font-size="36" font-weight="850">F</text><text x="507" y="472" class="boxtitle" text-anchor="middle">Filebeat</text><text x="507" y="504" class="meta" text-anchor="middle">tail • multiline • metadata</text><rect x="430" y="530" width="155" height="34" rx="17" class="pill"/><text x="507" y="552" class="pilltext" text-anchor="middle">disk registry</text></g>
  <g><rect x="705" y="340" width="225" height="250" class="box" stroke="#f5c142"/><circle cx="817" cy="405" r="38" fill="#876b10"/><text x="804" y="417" font-size="34" font-weight="850">L</text><text x="817" y="472" class="boxtitle" text-anchor="middle">Logstash</text><text x="817" y="504" class="meta" text-anchor="middle">parse • enrich • route</text><rect x="740" y="530" width="155" height="34" rx="17" class="pill"/><text x="817" y="552" class="pilltext" text-anchor="middle">persistent queue</text></g>
  <g><rect x="1025" y="275" width="310" height="390" class="box" stroke="#31b8dc"/><text x="1180" y="325" class="boxtitle" text-anchor="middle">Elasticsearch cluster</text><circle cx="1110" cy="420" r="66" fill="#123c58" stroke="#43c5e8" stroke-width="4"/><circle cx="1248" cy="420" r="66" fill="#123c58" stroke="#43c5e8" stroke-width="4"/><circle cx="1179" cy="540" r="66" fill="#123c58" stroke="#43c5e8" stroke-width="4"/><text x="1110" y="413" font-size="20" text-anchor="middle" font-weight="750">data-1</text><text x="1110" y="440" class="meta" text-anchor="middle">primary</text><text x="1248" y="413" font-size="20" text-anchor="middle" font-weight="750">data-2</text><text x="1248" y="440" class="meta" text-anchor="middle">replica</text><text x="1179" y="533" font-size="20" text-anchor="middle" font-weight="750">data-3</text><text x="1179" y="560" class="meta" text-anchor="middle">replica</text><rect x="1080" y="610" width="200" height="34" rx="17" class="pill"/><text x="1180" y="632" class="pilltext" text-anchor="middle">data streams • ILM</text></g>
  <g><rect x="1435" y="235" width="190" height="150" class="box" stroke="#e758a8"/><text x="1530" y="285" class="boxtitle" text-anchor="middle">Kibana</text><text x="1530" y="322" class="meta" text-anchor="middle">Discover / Lens</text><rect x="1470" y="340" width="120" height="30" rx="15" class="pill"/><text x="1530" y="361" class="pilltext" text-anchor="middle">TCP 5601</text></g>
  <g><rect x="1660" y="235" width="190" height="150" class="box" stroke="#ff9830"/><text x="1755" y="285" class="boxtitle" text-anchor="middle">Grafana</text><text x="1755" y="322" class="meta" text-anchor="middle">Explore / dashboards</text><rect x="1695" y="340" width="120" height="30" rx="15" class="pill"/><text x="1755" y="361" class="pilltext" text-anchor="middle">TCP 3000</text></g>
  <g><rect x="1495" y="625" width="295" height="150" class="box" stroke="#ff5f6d"/><text x="1642" y="675" class="boxtitle" text-anchor="middle">Notification channels</text><text x="1642" y="712" class="meta" text-anchor="middle">Slack • Email • PagerDuty</text><rect x="1560" y="730" width="165" height="30" rx="15" class="pill"/><text x="1642" y="751" class="pilltext" text-anchor="middle">webhook / SMTP</text></g>
  <path d="M315 300 C355 300 355 385 395 385" class="flow"/><path d="M315 495 H395" class="flow"/><path d="M315 690 C355 690 355 545 395 545" class="flow"/>
  <path d="M620 465 H705" class="flow"/><text x="662" y="442" class="label" text-anchor="middle">Beats/TCP</text><text x="662" y="462" class="label" text-anchor="middle">5044</text>
  <path d="M930 465 H1025" class="flow"/><text x="977" y="425" class="label" text-anchor="middle">Bulk API</text><text x="977" y="445" class="label" text-anchor="middle">HTTP(S) 9200</text>
  <path d="M1335 355 C1385 355 1385 310 1435 310" class="query"/><path d="M1335 500 C1450 500 1535 310 1660 310" class="query"/><text x="1393" y="482" class="label" text-anchor="middle">Search API :9200</text>
  <path d="M1755 385 C1755 505 1680 545 1642 625" class="alert"/><path d="M1530 385 C1530 505 1580 550 1615 625" class="alert"/><text x="1720" y="555" class="label" text-anchor="middle">alert rules</text>
  <rect x="70" y="945" width="1780" height="80" rx="18" fill="#0c1728" stroke="#263b58"/>
  <line x1="110" y1="985" x2="180" y2="985" class="flow"/><text x="200" y="991" class="meta">ingest / write path</text><line x1="465" y1="985" x2="535" y2="985" class="query"/><text x="555" y="991" class="meta">query + response</text><line x1="830" y1="985" x2="900" y2="985" class="alert"/><text x="920" y="991" class="meta">alert delivery</text><text x="1270" y="991" class="meta">Production: enable TLS/RBAC, snapshots, ≥3 ES nodes and external secrets</text>
 </svg>
@@ -0,0 +1,55 @@
 <svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080" viewBox="0 0 1920 1080">
  <defs>
    <linearGradient id="bg2" x1="0" y1="0" x2="1" y2="1"><stop stop-color="#08101d"/><stop offset="1" stop-color="#171326"/></linearGradient>
    <filter id="shadow2"><feDropShadow dx="0" dy="10" stdDeviation="12" flood-opacity=".25"/></filter>
    <marker id="mBlue" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#47b7ff"/></marker>
    <marker id="mGreen" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#55d187"/></marker>
    <marker id="mPurple" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#a987ff"/></marker>
    <marker id="mOrange" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#ff9a3d"/></marker>
    <marker id="mRed" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#ff5f6d"/></marker>
    <style>
      text{font-family:Inter,Segoe UI,Arial,sans-serif;fill:#edf4ff}.title{font-size:42px;font-weight:800}.sub{font-size:18px;fill:#9fb0c9}.layer{font-size:15px;font-weight:700;letter-spacing:2px;fill:#7185a3}.stage{fill:#0c1626;stroke:#273a56;stroke-width:1.5;rx:22}.box{fill:#142138;stroke:#324967;stroke-width:2;rx:18;filter:url(#shadow2)}.boxtitle{font-size:23px;font-weight:750}.meta{font-size:15px;fill:#b2c0d3}.pill{fill:#1e304b;stroke:#3d5878}.pilltext{font-size:14px;fill:#d9e8fb}.metric{fill:none;stroke:#55d187;stroke-width:4;marker-end:url(#mGreen)}.log{fill:none;stroke:#47b7ff;stroke-width:4;marker-end:url(#mBlue)}.trace{fill:none;stroke:#a987ff;stroke-width:4;marker-end:url(#mPurple)}.query{fill:none;stroke:#ff9a3d;stroke-width:4;stroke-dasharray:12 9;marker-start:url(#mOrange);marker-end:url(#mOrange)}.alert{fill:none;stroke:#ff5f6d;stroke-width:4;marker-end:url(#mRed)}.label{font-size:14px;font-weight:650;paint-order:stroke;stroke:#0a1424;stroke-width:7px;stroke-linejoin:round}
    </style>
  </defs>
  <rect width="1920" height="1080" fill="url(#bg2)"/>
  <text x="70" y="74" class="title">Grafana Observability Stack — Operational Data Flow</text>
  <text x="70" y="108" class="sub">Metrics (Prometheus) • logs (Loki) • traces (Tempo) • alert routing (Alertmanager) • unified exploration (Grafana)</text>
  <rect x="45" y="150" width="300" height="750" class="stage"/><text x="70" y="186" class="layer">WORKLOADS &amp; HOSTS</text>
  <rect x="370" y="150" width="300" height="750" class="stage"/><text x="395" y="186" class="layer">COLLECT &amp; EXPORT</text>
  <rect x="695" y="150" width="390" height="750" class="stage"/><text x="720" y="186" class="layer">STORE &amp; QUERY</text>
  <rect x="1110" y="150" width="295" height="750" class="stage"/><text x="1135" y="186" class="layer">ALERT PIPELINE</text>
  <rect x="1430" y="150" width="445" height="750" class="stage"/><text x="1455" y="186" class="layer">EXPLORE &amp; CORRELATE</text>
  <g><rect x="70" y="230" width="250" height="140" class="box"/><circle cx="110" cy="273" r="22" fill="#55d187"/><text x="102" y="281" font-size="22" font-weight="850">M</text><text x="148" y="275" class="boxtitle">App metrics</text><text x="95" y="318" class="meta">/metrics endpoint</text><text x="95" y="343" class="meta">Prometheus exposition</text></g>
  <g><rect x="70" y="425" width="250" height="140" class="box"/><circle cx="110" cy="468" r="22" fill="#47b7ff"/><text x="103" y="476" font-size="22" font-weight="850">L</text><text x="148" y="470" class="boxtitle">Container logs</text><text x="95" y="513" class="meta">Docker socket / files</text><text x="95" y="538" class="meta">structured log records</text></g>
  <g><rect x="70" y="620" width="250" height="140" class="box"/><circle cx="110" cy="663" r="22" fill="#a987ff"/><text x="103" y="671" font-size="22" font-weight="850">T</text><text x="148" y="665" class="boxtitle">Instrumented apps</text><text x="95" y="708" class="meta">OpenTelemetry SDK</text><text x="95" y="733" class="meta">spans + resource attrs</text></g>
  <g><rect x="400" y="270" width="240" height="290" class="box" stroke="#ff9a3d"/><circle cx="520" cy="342" r="42" fill="#a95014"/><text x="501" y="355" font-size="34" font-weight="850">A</text><text x="520" y="415" class="boxtitle" text-anchor="middle">Grafana Alloy</text><text x="520" y="450" class="meta" text-anchor="middle">discover • label • batch</text><rect x="435" y="478" width="170" height="34" rx="17" class="pill"/><text x="520" y="500" class="pilltext" text-anchor="middle">OTLP 4317 / 4318</text><rect x="455" y="520" width="130" height="27" rx="14" class="pill"/><text x="520" y="539" class="pilltext" text-anchor="middle">UI 12345</text></g>
  <g><rect x="400" y="635" width="240" height="150" class="box" stroke="#55d187"/><text x="520" y="685" class="boxtitle" text-anchor="middle">Node Exporter</text><text x="520" y="722" class="meta" text-anchor="middle">host CPU / RAM / disk</text><rect x="460" y="742" width="120" height="28" rx="14" class="pill"/><text x="520" y="762" class="pilltext" text-anchor="middle">TCP 9100</text></g>
  <g><rect x="730" y="220" width="320" height="170" class="box" stroke="#55d187"/><text x="890" y="270" class="boxtitle" text-anchor="middle">Prometheus</text><text x="890" y="308" class="meta" text-anchor="middle">scrape • TSDB • PromQL • rules</text><rect x="812" y="335" width="156" height="30" rx="15" class="pill"/><text x="890" y="356" class="pilltext" text-anchor="middle">HTTP 9090</text></g>
  <g><rect x="730" y="455" width="320" height="170" class="box" stroke="#47b7ff"/><text x="890" y="505" class="boxtitle" text-anchor="middle">Loki</text><text x="890" y="543" class="meta" text-anchor="middle">labels • chunks • LogQL</text><rect x="812" y="570" width="156" height="30" rx="15" class="pill"/><text x="890" y="591" class="pilltext" text-anchor="middle">HTTP 3100</text></g>
  <g><rect x="730" y="690" width="320" height="170" class="box" stroke="#a987ff"/><text x="890" y="740" class="boxtitle" text-anchor="middle">Tempo</text><text x="890" y="778" class="meta" text-anchor="middle">trace blocks • TraceQL</text><rect x="785" y="805" width="210" height="30" rx="15" class="pill"/><text x="890" y="826" class="pilltext" text-anchor="middle">query 3200 • OTLP 4317</text></g>
  <g><rect x="1145" y="300" width="225" height="200" class="box" stroke="#ff5f6d"/><text x="1257" y="355" class="boxtitle" text-anchor="middle">Alertmanager</text><text x="1257" y="393" class="meta" text-anchor="middle">group • dedupe • route</text><rect x="1182" y="420" width="150" height="30" rx="15" class="pill"/><text x="1257" y="441" class="pilltext" text-anchor="middle">HTTP 9093</text></g>
  <g><rect x="1145" y="610" width="225" height="150" class="box" stroke="#ff5f6d"/><text x="1257" y="660" class="boxtitle" text-anchor="middle">Receivers</text><text x="1257" y="700" class="meta" text-anchor="middle">Slack • Email • PagerDuty</text><text x="1257" y="730" class="meta" text-anchor="middle">webhook / SMTP</text></g>
  <g><rect x="1470" y="300" width="365" height="350" class="box" stroke="#ff9a3d"/><circle cx="1652" cy="388" r="50" fill="#ba5a14"/><text x="1631" y="404" font-size="42" font-weight="850">G</text><text x="1652" y="477" class="boxtitle" text-anchor="middle">Grafana</text><text x="1652" y="518" class="meta" text-anchor="middle">dashboards • Explore • correlations</text><rect x="1547" y="548" width="210" height="34" rx="17" class="pill"/><text x="1652" y="570" class="pilltext" text-anchor="middle">Metrics ↔ Logs ↔ Traces</text><rect x="1592" y="596" width="120" height="30" rx="15" class="pill"/><text x="1652" y="617" class="pilltext" text-anchor="middle">TCP 3000</text></g>
  <path d="M320 300 C360 300 360 315 400 315" class="metric"/><text x="354" y="286" class="label" text-anchor="middle">scrape</text>
  <path d="M320 495 H400" class="log"/><text x="360" y="480" class="label" text-anchor="middle">tail</text>
  <path d="M320 690 C360 690 360 515 400 515" class="trace"/><text x="355" y="655" class="label" text-anchor="middle">OTLP</text>
  <path d="M640 350 C680 350 690 305 730 305" class="metric"/><text x="684" y="292" class="label" text-anchor="middle">/metrics</text>
  <path d="M640 485 C680 485 690 540 730 540" class="log"/><text x="685" y="513" class="label" text-anchor="middle">push API</text>
  <path d="M640 520 C690 520 680 775 730 775" class="trace"/><text x="681" y="728" class="label" text-anchor="middle">OTLP/gRPC</text>
  <path d="M640 710 C690 710 680 340 730 340" class="metric"/><text x="670" y="630" class="label" text-anchor="middle">scrape :9100</text>
  <path d="M1050 305 H1145" class="alert"/><text x="1097" y="286" class="label" text-anchor="middle">firing alerts</text>
  <path d="M1257 500 V610" class="alert"/><text x="1305" y="560" class="label">route</text>
  <path d="M1050 305 C1210 210 1370 260 1470 365" class="query"/><path d="M1050 540 C1210 540 1340 470 1470 470" class="query"/><path d="M1050 775 C1220 775 1350 625 1470 575" class="query"/><text x="1300" y="452" class="label" text-anchor="middle">PromQL / LogQL / TraceQL</text>
  <path d="M1050 760 C1090 720 1090 390 1050 350" class="metric"/><text x="1092" y="560" class="label" text-anchor="middle" transform="rotate(-90 1092 560)">span metrics + exemplars → remote_write</text>
  <rect x="70" y="945" width="1780" height="80" rx="18" fill="#0c1728" stroke="#263b58"/>
  <line x1="105" y1="982" x2="170" y2="982" class="metric"/><text x="190" y="988" class="meta">metrics</text><line x1="340" y1="982" x2="405" y2="982" class="log"/><text x="425" y="988" class="meta">logs</text><line x1="555" y1="982" x2="620" y2="982" class="trace"/><text x="640" y="988" class="meta">traces</text><line x1="780" y1="982" x2="845" y2="982" class="query"/><text x="865" y="988" class="meta">query + response</text><line x1="1080" y1="982" x2="1145" y2="982" class="alert"/><text x="1165" y="988" class="meta">alerts</text><text x="1390" y="988" class="meta">Correlation key: trace_id / exemplars</text>
 </svg>
@@ -0,0 +1,29 @@
 ---
 # Deployment base path on the target host.
 observability_root: /opt/observability
 # Change these tags deliberately when upgrading.
 elastic_version: "9.1.3"
 grafana_version: "12.1.1"
 prometheus_version: "v3.5.0"
 alertmanager_version: "v0.28.1"
 loki_version: "3.5.3"
 tempo_version: "2.8.2"
 alloy_version: "v1.10.2"
 node_exporter_version: "v1.9.1"
 # Lab-safe defaults. Override with Ansible Vault for real deployments.
 grafana_admin_user: admin
 grafana_admin_password: "change-me-now"
 # Optional Alertmanager webhook. Leave empty to install a no-op receiver.
 alertmanager_webhook_url: ""
 # Add application /metrics endpoints here. Example:
 # prometheus_extra_scrape_jobs:
 #   - job_name: checkout-api
 #     targets: ["10.20.0.15:8080", "10.20.0.16:8080"]
 prometheus_extra_scrape_jobs: []
 # JVM heap for single-node Elasticsearch. Keep Xms and Xmx equal.
 elasticsearch_heap: 1g
@@ -0,0 +1,6 @@
 ---
 all:
  children:
    observability:
      hosts:
        cloudzy:
@@ -0,0 +1,10 @@
 ---
 - name: Deploy ELK and Grafana
  hosts: observability
  become: true
  vars:
    observability_stack: elk-grafana
  roles:
    - docker
    - observability
@@ -0,0 +1,10 @@
 ---
 - name: Deploy Grafana observability stack
  hosts: observability
  become: true
  vars:
    observability_stack: grafana-stack
  roles:
    - docker
    - observability
@@ -0,0 +1,7 @@
 ---
 collections:
  - name: community.docker
    version: ">=4.0.0"
  - name: ansible.posix
    version: ">=2.0.0"
@@ -0,0 +1,56 @@
 ---
 - name: Assert supported operating system
  ansible.builtin.assert:
    that:
      - ansible_facts.os_family == 'Debian'
      - ansible_facts.distribution in ['Ubuntu', 'Debian']
    fail_msg: This role supports Ubuntu and Debian hosts.
 - name: Install Docker repository prerequisites
  ansible.builtin.apt:
    name:
      - ca-certificates
      - curl
      - gnupg
    state: present
    update_cache: true
 - name: Create APT keyrings directory
  ansible.builtin.file:
    path: /etc/apt/keyrings
    state: directory
    mode: "0755"
 - name: Download Docker signing key
  ansible.builtin.get_url:
    url: "https://download.docker.com/linux/{{ ansible_facts.distribution | lower }}/gpg"
    dest: /etc/apt/keyrings/docker.asc
    mode: "0644"
 - name: Configure Docker APT repository
  ansible.builtin.apt_repository:
    repo: >-
      deb [arch={{ {'x86_64':'amd64','aarch64':'arm64'}.get(ansible_facts.architecture, ansible_facts.architecture) }}
      signed-by=/etc/apt/keyrings/docker.asc]
      https://download.docker.com/linux/{{ ansible_facts.distribution | lower }}
      {{ ansible_facts.distribution_release }} stable
    filename: docker
    state: present
 - name: Install Docker Engine and Compose plugin
  ansible.builtin.apt:
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin
    state: present
    update_cache: true
 - name: Enable and start Docker
  ansible.builtin.systemd_service:
    name: docker
    state: started
    enabled: true
@@ -0,0 +1,7 @@
 ---
 - name: Restart observability stack
  community.docker.docker_compose_v2:
    project_src: "{{ stack_root }}"
    state: restarted
  when: stack_root is defined
@@ -0,0 +1,90 @@
 ---
 - name: Validate selected stack
  ansible.builtin.assert:
    that:
      - observability_stack in ['elk-grafana', 'grafana-stack']
    fail_msg: observability_stack must be elk-grafana or grafana-stack.
 - name: Set stack deployment directory
  ansible.builtin.set_fact:
    stack_root: "{{ observability_root }}/{{ observability_stack }}"
 - name: Create stack directory tree
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: root
    group: root
    mode: "0755"
  loop: >-
    {{
      [stack_root, stack_root + '/config']
      + (
          [stack_root + '/config/filebeat', stack_root + '/config/logstash/pipeline', stack_root + '/config/grafana/provisioning/datasources']
          if observability_stack == 'elk-grafana'
          else [stack_root + '/config/prometheus/rules', stack_root + '/config/alertmanager', stack_root + '/config/loki', stack_root + '/config/tempo', stack_root + '/config/alloy', stack_root + '/config/grafana/provisioning/datasources']
        )
    }}
 - name: Apply Elasticsearch kernel requirement
  ansible.posix.sysctl:
    name: vm.max_map_count
    value: "262144"
    state: present
    sysctl_set: true
    reload: true
  when: observability_stack == 'elk-grafana'
 - name: Render ELK and Grafana files
  ansible.builtin.template:
    src: "elk-grafana/{{ item.src }}"
    dest: "{{ stack_root }}/{{ item.dest }}"
    owner: root
    group: root
    mode: "{{ item.mode | default('0644') }}"
  loop:
    - { src: docker-compose.yml.j2, dest: docker-compose.yml }
    - { src: config/filebeat/filebeat.yml.j2, dest: config/filebeat/filebeat.yml }
    - { src: config/logstash/pipeline/logstash.conf.j2, dest: config/logstash/pipeline/logstash.conf }
    - { src: config/grafana/provisioning/datasources/datasources.yml.j2, dest: config/grafana/provisioning/datasources/datasources.yml }
  when: observability_stack == 'elk-grafana'
  notify: Restart observability stack
 - name: Render Grafana stack files
  ansible.builtin.template:
    src: "grafana-stack/{{ item.src }}"
    dest: "{{ stack_root }}/{{ item.dest }}"
    owner: root
    group: root
    mode: "{{ item.mode | default('0644') }}"
  loop:
    - { src: docker-compose.yml.j2, dest: docker-compose.yml }
    - { src: config/prometheus/prometheus.yml.j2, dest: config/prometheus/prometheus.yml }
    - { src: config/prometheus/rules/host.rules.yml.j2, dest: config/prometheus/rules/host.rules.yml }
    - { src: config/alertmanager/alertmanager.yml.j2, dest: config/alertmanager/alertmanager.yml }
    - { src: config/loki/loki.yml.j2, dest: config/loki/loki.yml }
    - { src: config/tempo/tempo.yml.j2, dest: config/tempo/tempo.yml }
    - { src: config/alloy/config.alloy.j2, dest: config/alloy/config.alloy }
    - { src: config/grafana/provisioning/datasources/datasources.yml.j2, dest: config/grafana/provisioning/datasources/datasources.yml }
  when: observability_stack == 'grafana-stack'
  notify: Restart observability stack
 - name: Start selected observability stack
  community.docker.docker_compose_v2:
    project_src: "{{ stack_root }}"
    state: present
    pull: missing
    remove_orphans: true
 - name: Flush restart handler before health checks
  ansible.builtin.meta: flush_handlers
 - name: Wait for Grafana
  ansible.builtin.uri:
    url: http://127.0.0.1:3000/api/health
    status_code: 200
  register: grafana_health
  retries: 30
  delay: 5
  until: grafana_health.status == 200
@@ -0,0 +1,21 @@
 filebeat.inputs:
  - type: filestream
    id: docker-containers
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log
    parsers:
      - container:
          stream: all
          format: docker
 processors:
  - add_docker_metadata:
      host: unix:///var/run/docker.sock
  - add_host_metadata: ~
 output.logstash:
  hosts: ["logstash:5044"]
 logging.level: info
@@ -0,0 +1,18 @@
 apiVersion: 1
 datasources:
  - name: Elasticsearch
    uid: elasticsearch
    type: elasticsearch
    access: proxy
    url: http://elasticsearch:9200
    database: "logs-*"
    isDefault: true
    editable: true
    jsonData:
      index: "logs-*"
      interval: Daily
      timeField: "@timestamp"
      logMessageField: message
      logLevelField: log.level
@@ -0,0 +1,30 @@
 input {
  beats {
    port => 5044
  }
  tcp {
    port => 5000
    codec => json_lines
  }
 }
 filter {
  if [message] =~ /^\s*\{/ {
    json {
      source => "message"
      skip_on_invalid_json => true
    }
  }
  mutate {
    add_field => { "[@metadata][target_index]" => "logs-%{+YYYY.MM.dd}" }
  }
 }
 output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "%{[@metadata][target_index]}"
  }
  stdout { codec => dots }
 }
@@ -0,0 +1,86 @@
 services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:{{ elastic_version }}
    restart: unless-stopped
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
      xpack.security.enrollment.enabled: "false"
      ES_JAVA_OPTS: "-Xms{{ elasticsearch_heap }} -Xmx{{ elasticsearch_heap }}"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 30
    networks: [observability]
  logstash:
    image: docker.elastic.co/logstash/logstash:{{ elastic_version }}
    restart: unless-stopped
    depends_on:
      elasticsearch:
        condition: service_healthy
    volumes:
      - ./config/logstash/pipeline:/usr/share/logstash/pipeline:ro
      - logstash-data:/usr/share/logstash/data
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
    networks: [observability]
  filebeat:
    image: docker.elastic.co/beats/filebeat:{{ elastic_version }}
    user: root
    restart: unless-stopped
    command: ["filebeat", "-e", "--strict.perms=false"]
    depends_on: [logstash]
    volumes:
      - ./config/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - filebeat-data:/usr/share/filebeat/data
    networks: [observability]
  kibana:
    image: docker.elastic.co/kibana/kibana:{{ elastic_version }}
    restart: unless-stopped
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
      - "5601:5601"
    networks: [observability]
  grafana:
    image: grafana/grafana:{{ grafana_version }}
    restart: unless-stopped
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      GF_SECURITY_ADMIN_USER: "{{ grafana_admin_user }}"
      GF_SECURITY_ADMIN_PASSWORD: "{{ grafana_admin_password }}"
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - grafana-data:/var/lib/grafana
      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    networks: [observability]
 volumes:
  elasticsearch-data:
  logstash-data:
  filebeat-data:
  grafana-data:
 networks:
  observability:
    driver: bridge
@@ -0,0 +1,18 @@
 global:
  resolve_timeout: 5m
 route:
  receiver: default
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
 receivers:
  - name: default
 {% if alertmanager_webhook_url | length > 0 %}
    webhook_configs:
      - url: "{{ alertmanager_webhook_url }}"
        send_resolved: true
 {% endif %}
@@ -0,0 +1,51 @@
 logging {
  level = "info"
 }
 discovery.docker "local" {
  host = "unix:///var/run/docker.sock"
 }
 discovery.relabel "docker" {
  targets = discovery.docker.local.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
 }
 loki.source.docker "local" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.docker.output
  forward_to = [loki.write.default.receiver]
 }
 loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
 }
 otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
 }
 otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls {
      insecure = true
    }
  }
 }
@@ -0,0 +1,54 @@
 apiVersion: 1
 datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
    jsonData:
      exemplarTraceIdDestinations:
        - datasourceUid: tempo
          name: trace_id
  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: true
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
          name: TraceID
          url: '$${__value.raw}'
  - name: Tempo
    uid: tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: true
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        spanStartTimeShift: -1m
        spanEndTimeShift: 1m
      serviceMap:
        datasourceUid: prometheus
      nodeGraph:
        enabled: true
  - name: Alertmanager
    uid: alertmanager
    type: alertmanager
    access: proxy
    url: http://alertmanager:9093
    editable: true
    jsonData:
      implementation: prometheus
@@ -0,0 +1,34 @@
 auth_enabled: false
 server:
  http_listen_port: 3100
 common:
  path_prefix: /var/loki
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
 schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
 storage_config:
  filesystem:
    directory: /var/loki/chunks
 compactor:
  working_directory: /var/loki/compactor
 limits_config:
  retention_period: 336h
  allow_structured_metadata: true
@@ -0,0 +1,43 @@
 global:
  scrape_interval: 15s
  evaluation_interval: 15s
 rule_files:
  - /etc/prometheus/rules/*.yml
 alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
 scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["prometheus:9090"]
  - job_name: node-exporter
    static_configs:
      - targets: ["node-exporter:9100"]
  - job_name: loki
    static_configs:
      - targets: ["loki:3100"]
  - job_name: tempo
    static_configs:
      - targets: ["tempo:3200"]
  - job_name: alloy
    static_configs:
      - targets: ["alloy:12345"]
  - job_name: grafana
    static_configs:
      - targets: ["grafana:3000"]
 {% for job in prometheus_extra_scrape_jobs %}
  - job_name: "{{ job.job_name }}"
    static_configs:
      - targets: {{ job.targets | to_json }}
 {% endfor %}
@@ -0,0 +1,21 @@
 groups:
  - name: host.rules
    rules:
      - alert: HostExporterDown
        expr: up{job="node-exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node exporter is unavailable"
          description: "Prometheus has failed to scrape {{ '{{' }} $labels.instance {{ '}}' }} for more than two minutes."
      - alert: HostHighCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU utilization"
          description: "CPU utilization on {{ '{{' }} $labels.instance {{ '}}' }} has exceeded 85% for ten minutes."
@@ -0,0 +1,42 @@
 server:
  http_listen_port: 3200
 distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
 ingester:
  max_block_duration: 5m
 compactor:
  compaction:
    block_retention: 336h
 storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks
 metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
 overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]
@@ -0,0 +1,110 @@
 services:
  prometheus:
    image: prom/prometheus:{{ prometheus_version }}
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=15d
      - --web.enable-lifecycle
      - --web.enable-remote-write-receiver
    volumes:
      - ./config/prometheus:/etc/prometheus:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    networks: [observability]
  alertmanager:
    image: prom/alertmanager:{{ alertmanager_version }}
    restart: unless-stopped
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./config/alertmanager:/etc/alertmanager:ro
      - alertmanager-data:/alertmanager
    ports:
      - "9093:9093"
    networks: [observability]
  loki:
    image: grafana/loki:{{ loki_version }}
    restart: unless-stopped
    command: ["-config.file=/etc/loki/loki.yml"]
    volumes:
      - ./config/loki/loki.yml:/etc/loki/loki.yml:ro
      - loki-data:/var/loki
    ports:
      - "3100:3100"
    networks: [observability]
  tempo:
    image: grafana/tempo:{{ tempo_version }}
    restart: unless-stopped
    command: ["-config.file=/etc/tempo/tempo.yml"]
    volumes:
      - ./config/tempo/tempo.yml:/etc/tempo/tempo.yml:ro
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"
    networks: [observability]
  node-exporter:
    image: prom/node-exporter:{{ node_exporter_version }}
    restart: unless-stopped
    command:
      - --path.rootfs=/host
    pid: host
    volumes:
      - /:/host:ro,rslave
    ports:
      - "9100:9100"
    networks: [observability]
  alloy:
    image: grafana/alloy:{{ alloy_version }}
    restart: unless-stopped
    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config.alloy
    volumes:
      - ./config/alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - alloy-data:/var/lib/alloy/data
    ports:
      - "12345:12345"
      - "4317:4317"
      - "4318:4318"
    networks: [observability]
  grafana:
    image: grafana/grafana:{{ grafana_version }}
    restart: unless-stopped
    depends_on: [prometheus, loki, tempo]
    environment:
      GF_SECURITY_ADMIN_USER: "{{ grafana_admin_user }}"
      GF_SECURITY_ADMIN_PASSWORD: "{{ grafana_admin_password }}"
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_FEATURE_TOGGLES_ENABLE: traceqlEditor
    volumes:
      - grafana-data:/var/lib/grafana
      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    networks: [observability]
 volumes:
  prometheus-data:
  alertmanager-data:
  loki-data:
  tempo-data:
  alloy-data:
  grafana-data:
 networks:
  observability:
    driver: bridge