diff --git a/ansible/README.md b/ansible/README.md
new file mode 100644
index 0000000..189eb23
--- /dev/null
+++ b/ansible/README.md
@@ -0,0 +1,204 @@
+# Deploying Two Observability Stacks with Ansible
+
+This project deploys two independent stacks on a single Ubuntu/Debian host using Docker Compose:
+
+1. **ELK + Grafana:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana
+2. **Grafana Observability:** Prometheus + Alertmanager + Loki + Tempo + Grafana, along with Alloy and Node Exporter for data collection
+
+> The default settings are suitable for Lab, Demo, and training purposes. For Production, be sure to read the "Production Notes" section.
+
+## Operational Diagrams
+
+- `diagrams/01-elk-grafana-operational.png` and its SVG vector version
+- `diagrams/02-grafana-lgtm-operational.png` and its SVG vector version
+
+Path colors in the second diagram:
+
+- Green: Metrics
+- Blue: Logs
+- Purple: Traces
+- Dashed orange: Query/Response
+- Red: Alert delivery
+
+## Project Structure
+
+```text
+inventory/hosts.yml SSH targets
+group_vars/all.yml Versions, Grafana password, and variables
+playbooks/elk-grafana.yml Installs ELK + Grafana
+playbooks/grafana-stack.yml Installs Grafana/Prometheus/Loki/Tempo
+roles/docker/ Installs Docker Engine and Compose v2
+roles/observability/ Templates, execution, and health checks
+diagrams/ PNG and SVG diagrams
+```
+
+## Prerequisites
+
+- Control node: Ansible Core 2.16 or newer
+- Target: Ubuntu 22.04/24.04 or Debian 12, with SSH and sudo access
+- Recommended minimum for the ELK Lab: 4 vCPUs, 8 GB RAM, and 50 GB disk
+- Recommended minimum for the Grafana Stack Lab: 2 vCPUs, 4 GB RAM, and 30 GB disk
+
+On the control node:
+
+```bash
+cd observability-ansible
+ansible-galaxy collection install -r requirements.yml
+```
+
+## 1. Configure the Inventory
+
+Edit the `inventory/hosts.yml` file:
+
+```yaml
+all:
+ children:
+ observability:
+ hosts:
+ obs01:
+ ansible_host: 10.10.20.15
+ ansible_user: ubuntu
+ ansible_ssh_private_key_file: ~/.ssh/id_ed25519
+```
+
+Test the connection:
+
+```bash
+ansible observability -m ping
+```
+
+## 2. Configure Variables and Secrets
+
+Versions and core settings live in `group_vars/all.yml`. Be sure to change the default Grafana password. To store the password securely:
+
+```bash
+ansible-vault encrypt_string 'A-strong-password' --name grafana_admin_password
+```
+
+Replace the `grafana_admin_password` value with the Vault output, and run the playbook with `--ask-vault-pass`.
+
+If your application exposes a `/metrics` endpoint, add it here:
+
+```yaml
+prometheus_extra_scrape_jobs:
+ - job_name: checkout-api
+ targets:
+ - 10.20.0.15:8080
+ - 10.20.0.16:8080
+```
+
+To send alerts to a webhook, also set the following value:
+
+```yaml
+alertmanager_webhook_url: "https://example.internal/alerts"
+```
+
+## 3. Install ELK + Grafana
+
+```bash
+ansible-playbook playbooks/elk-grafana.yml
+```
+
+Endpoints:
+
+| Service | URL/Port | Purpose |
+| --- | --- | --- |
+| Elasticsearch | `http://SERVER:9200` | Index and Search API |
+| Logstash Beats input | `SERVER:5044` | Receives Filebeat |
+| Logstash JSON/TCP | `SERVER:5000` | Testing and direct JSON Lines ingest |
+| Kibana | `http://SERVER:5601` | Discover and Elastic analysis |
+| Grafana | `http://SERVER:3000` | Dashboards on top of Elasticsearch |
+
+Filebeat in this project reads the Docker container logs on the same host and sends them to Logstash. Logstash parses the JSON and writes it into the daily `logs-YYYY.MM.dd` index.
+
+Test ingest:
+
+```bash
+printf '%s\n' '{"message":"checkout completed","level":"info","service":"checkout"}' \
+ | nc SERVER 5000
+
+curl 'http://SERVER:9200/logs-*/_search?pretty'
+```
+
+## 4. Install Grafana + Prometheus + Alertmanager + Loki + Tempo
+
+```bash
+ansible-playbook playbooks/grafana-stack.yml
+```
+
+Endpoints:
+
+| Service | URL/Port | Purpose |
+| --- | --- | --- |
+| Grafana | `http://SERVER:3000` | Explore, Dashboards, and correlation |
+| Prometheus | `http://SERVER:9090` | Metrics, PromQL, and alert rules |
+| Alertmanager | `http://SERVER:9093` | Grouping, deduplication, and routing of alerts |
+| Loki | `http://SERVER:3100` | Log ingestion/query and LogQL |
+| Tempo | `http://SERVER:3200` | Trace query and TraceQL |
+| Alloy UI | `http://SERVER:12345` | Collector pipeline status |
+| OTLP/gRPC | `SERVER:4317` | Receives traces from the application |
+| OTLP/HTTP | `SERVER:4318` | Receives traces from the application |
+| Node Exporter | `http://SERVER:9100/metrics` | Host-level metrics |
+
+The Prometheus, Loki, Tempo, and Alertmanager data sources are automatically provisioned in Grafana. Grafana uses exemplars for Metrics → Trace, `trace_id` for Logs → Trace, and Prometheus for the Tempo service graph.
+
+Example OpenTelemetry environment variables for your application:
+
+```bash
+export OTEL_SERVICE_NAME=checkout-api
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://SERVER:4318
+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+```
+
+Alloy pushes Docker container logs to Loki and forwards OTLP traces to Tempo. Prometheus scrapes internal services and Node Exporter. The sample rules `HostExporterDown` and `HostHighCPU` are also installed.
+
+## Day-to-Day Operations
+
+On the target server:
+
+```bash
+cd /opt/observability/elk-grafana
+docker compose ps
+docker compose logs -f --tail=100
+
+cd /opt/observability/grafana-stack
+docker compose ps
+docker compose logs -f --tail=100
+```
+
+Re-running the playbook is idempotent; changing a template or version tag triggers a recreate/restart of the relevant stack. Data is kept in Docker named volumes.
+
+## Removing a Stack
+
+To stop and remove the containers without deleting the data:
+
+```bash
+cd /opt/observability/STACK_NAME
+docker compose down
+```
+
+The `-v` flag deletes all volumes and monitoring data; only use it when you have a backup and specifically intend to do this.
+
+## Production Notes
+
+The default playbook intentionally builds a single-node, easy-to-understand deployment. For Production:
+
+- Enable TLS, authentication, and RBAC on all endpoints; don't expose storage ports directly to the internet.
+- Keep secrets in Ansible Vault or a Secret Manager.
+- Run Elasticsearch with at least three nodes, a proper snapshot repository, and ILM.
+- Use S3-compatible object storage and replication for Loki and Tempo.
+- Design Prometheus with remote-write/Thanos/Mimir, retention, and adequate disk capacity.
+- Run Alertmanager in HA and test your notification routes.
+- Add a reverse proxy, DNS, certificates, firewall rules, and backup/restore testing.
+- Upgrade versions in staging first; they are intentionally pinned in `group_vars/all.yml`.
+
+## Official Resources
+
+- [Elastic Docker deployment](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-docker-compose)
+- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
+- [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
+- [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/)
+- [Grafana Loki configuration](https://grafana.com/docs/loki/latest/configure/)
+- [Grafana Tempo configuration](https://grafana.com/docs/tempo/latest/configuration/)
+- [Grafana Alloy](https://grafana.com/docs/alloy/latest/)
+- [Ansible Docker Compose v2 module](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_module.html)
diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg
new file mode 100644
index 0000000..2f7216e
--- /dev/null
+++ b/ansible/ansible.cfg
@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventory/hosts.yml
+roles_path = roles
+host_key_checking = False
+interpreter_python = auto_silent
+retry_files_enabled = False
+
+[ssh_connection]
+pipelining = True
+
diff --git a/ansible/diagrams/01-elk-grafana-operational.png b/ansible/diagrams/01-elk-grafana-operational.png
new file mode 100644
index 0000000..a82d3af
Binary files /dev/null and b/ansible/diagrams/01-elk-grafana-operational.png differ
diff --git a/ansible/diagrams/01-elk-grafana-operational.svg b/ansible/diagrams/01-elk-grafana-operational.svg
new file mode 100644
index 0000000..23ae755
--- /dev/null
+++ b/ansible/diagrams/01-elk-grafana-operational.svg
@@ -0,0 +1,44 @@
+
+
diff --git a/ansible/diagrams/02-grafana-lgtm-operational.png b/ansible/diagrams/02-grafana-lgtm-operational.png
new file mode 100644
index 0000000..4a988f4
Binary files /dev/null and b/ansible/diagrams/02-grafana-lgtm-operational.png differ
diff --git a/ansible/diagrams/02-grafana-lgtm-operational.svg b/ansible/diagrams/02-grafana-lgtm-operational.svg
new file mode 100644
index 0000000..4eadbd0
--- /dev/null
+++ b/ansible/diagrams/02-grafana-lgtm-operational.svg
@@ -0,0 +1,55 @@
+
+
diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml
new file mode 100644
index 0000000..43ae61f
--- /dev/null
+++ b/ansible/group_vars/all.yml
@@ -0,0 +1,29 @@
+---
+# Deployment base path on the target host.
+observability_root: /opt/observability
+
+# Change these tags deliberately when upgrading.
+elastic_version: "9.1.3"
+grafana_version: "12.1.1"
+prometheus_version: "v3.5.0"
+alertmanager_version: "v0.28.1"
+loki_version: "3.5.3"
+tempo_version: "2.8.2"
+alloy_version: "v1.10.2"
+node_exporter_version: "v1.9.1"
+
+# Lab-safe defaults. Override with Ansible Vault for real deployments.
+grafana_admin_user: admin
+grafana_admin_password: "change-me-now"
+
+# Optional Alertmanager webhook. Leave empty to install a no-op receiver.
+alertmanager_webhook_url: ""
+
+# Add application /metrics endpoints here. Example:
+# prometheus_extra_scrape_jobs:
+# - job_name: checkout-api
+# targets: ["10.20.0.15:8080", "10.20.0.16:8080"]
+prometheus_extra_scrape_jobs: []
+
+# JVM heap for single-node Elasticsearch. Keep Xms and Xmx equal.
+elasticsearch_heap: 1g
diff --git a/ansible/inventory/hosts.yml b/ansible/inventory/hosts.yml
new file mode 100644
index 0000000..de7466c
--- /dev/null
+++ b/ansible/inventory/hosts.yml
@@ -0,0 +1,6 @@
+---
+all:
+ children:
+ observability:
+ hosts:
+ cloudzy:
diff --git a/ansible/playbooks/elk-grafana.yml b/ansible/playbooks/elk-grafana.yml
new file mode 100644
index 0000000..1db6491
--- /dev/null
+++ b/ansible/playbooks/elk-grafana.yml
@@ -0,0 +1,10 @@
+---
+- name: Deploy ELK and Grafana
+ hosts: observability
+ become: true
+ vars:
+ observability_stack: elk-grafana
+ roles:
+ - docker
+ - observability
+
diff --git a/ansible/playbooks/grafana-stack.yml b/ansible/playbooks/grafana-stack.yml
new file mode 100644
index 0000000..443aec1
--- /dev/null
+++ b/ansible/playbooks/grafana-stack.yml
@@ -0,0 +1,10 @@
+---
+- name: Deploy Grafana observability stack
+ hosts: observability
+ become: true
+ vars:
+ observability_stack: grafana-stack
+ roles:
+ - docker
+ - observability
+
diff --git a/ansible/requirements.yml b/ansible/requirements.yml
new file mode 100644
index 0000000..79ac3be
--- /dev/null
+++ b/ansible/requirements.yml
@@ -0,0 +1,7 @@
+---
+collections:
+ - name: community.docker
+ version: ">=4.0.0"
+ - name: ansible.posix
+ version: ">=2.0.0"
+
diff --git a/ansible/roles/docker/tasks/main.yml b/ansible/roles/docker/tasks/main.yml
new file mode 100644
index 0000000..3c99014
--- /dev/null
+++ b/ansible/roles/docker/tasks/main.yml
@@ -0,0 +1,56 @@
+---
+- name: Assert supported operating system
+ ansible.builtin.assert:
+ that:
+ - ansible_facts.os_family == 'Debian'
+ - ansible_facts.distribution in ['Ubuntu', 'Debian']
+ fail_msg: This role supports Ubuntu and Debian hosts.
+
+- name: Install Docker repository prerequisites
+ ansible.builtin.apt:
+ name:
+ - ca-certificates
+ - curl
+ - gnupg
+ state: present
+ update_cache: true
+
+- name: Create APT keyrings directory
+ ansible.builtin.file:
+ path: /etc/apt/keyrings
+ state: directory
+ mode: "0755"
+
+- name: Download Docker signing key
+ ansible.builtin.get_url:
+ url: "https://download.docker.com/linux/{{ ansible_facts.distribution | lower }}/gpg"
+ dest: /etc/apt/keyrings/docker.asc
+ mode: "0644"
+
+- name: Configure Docker APT repository
+ ansible.builtin.apt_repository:
+ repo: >-
+ deb [arch={{ {'x86_64':'amd64','aarch64':'arm64'}.get(ansible_facts.architecture, ansible_facts.architecture) }}
+ signed-by=/etc/apt/keyrings/docker.asc]
+ https://download.docker.com/linux/{{ ansible_facts.distribution | lower }}
+ {{ ansible_facts.distribution_release }} stable
+ filename: docker
+ state: present
+
+- name: Install Docker Engine and Compose plugin
+ ansible.builtin.apt:
+ name:
+ - docker-ce
+ - docker-ce-cli
+ - containerd.io
+ - docker-buildx-plugin
+ - docker-compose-plugin
+ state: present
+ update_cache: true
+
+- name: Enable and start Docker
+ ansible.builtin.systemd_service:
+ name: docker
+ state: started
+ enabled: true
+
diff --git a/ansible/roles/observability/handlers/main.yml b/ansible/roles/observability/handlers/main.yml
new file mode 100644
index 0000000..861a259
--- /dev/null
+++ b/ansible/roles/observability/handlers/main.yml
@@ -0,0 +1,7 @@
+---
+- name: Restart observability stack
+ community.docker.docker_compose_v2:
+ project_src: "{{ stack_root }}"
+ state: restarted
+ when: stack_root is defined
+
diff --git a/ansible/roles/observability/tasks/main.yml b/ansible/roles/observability/tasks/main.yml
new file mode 100644
index 0000000..2de7f44
--- /dev/null
+++ b/ansible/roles/observability/tasks/main.yml
@@ -0,0 +1,90 @@
+---
+- name: Validate selected stack
+ ansible.builtin.assert:
+ that:
+ - observability_stack in ['elk-grafana', 'grafana-stack']
+ fail_msg: observability_stack must be elk-grafana or grafana-stack.
+
+- name: Set stack deployment directory
+ ansible.builtin.set_fact:
+ stack_root: "{{ observability_root }}/{{ observability_stack }}"
+
+- name: Create stack directory tree
+ ansible.builtin.file:
+ path: "{{ item }}"
+ state: directory
+ owner: root
+ group: root
+ mode: "0755"
+ loop: >-
+ {{
+ [stack_root, stack_root + '/config']
+ + (
+ [stack_root + '/config/filebeat', stack_root + '/config/logstash/pipeline', stack_root + '/config/grafana/provisioning/datasources']
+ if observability_stack == 'elk-grafana'
+ else [stack_root + '/config/prometheus/rules', stack_root + '/config/alertmanager', stack_root + '/config/loki', stack_root + '/config/tempo', stack_root + '/config/alloy', stack_root + '/config/grafana/provisioning/datasources']
+ )
+ }}
+
+- name: Apply Elasticsearch kernel requirement
+ ansible.posix.sysctl:
+ name: vm.max_map_count
+ value: "262144"
+ state: present
+ sysctl_set: true
+ reload: true
+ when: observability_stack == 'elk-grafana'
+
+- name: Render ELK and Grafana files
+ ansible.builtin.template:
+ src: "elk-grafana/{{ item.src }}"
+ dest: "{{ stack_root }}/{{ item.dest }}"
+ owner: root
+ group: root
+ mode: "{{ item.mode | default('0644') }}"
+ loop:
+ - { src: docker-compose.yml.j2, dest: docker-compose.yml }
+ - { src: config/filebeat/filebeat.yml.j2, dest: config/filebeat/filebeat.yml }
+ - { src: config/logstash/pipeline/logstash.conf.j2, dest: config/logstash/pipeline/logstash.conf }
+ - { src: config/grafana/provisioning/datasources/datasources.yml.j2, dest: config/grafana/provisioning/datasources/datasources.yml }
+ when: observability_stack == 'elk-grafana'
+ notify: Restart observability stack
+
+- name: Render Grafana stack files
+ ansible.builtin.template:
+ src: "grafana-stack/{{ item.src }}"
+ dest: "{{ stack_root }}/{{ item.dest }}"
+ owner: root
+ group: root
+ mode: "{{ item.mode | default('0644') }}"
+ loop:
+ - { src: docker-compose.yml.j2, dest: docker-compose.yml }
+ - { src: config/prometheus/prometheus.yml.j2, dest: config/prometheus/prometheus.yml }
+ - { src: config/prometheus/rules/host.rules.yml.j2, dest: config/prometheus/rules/host.rules.yml }
+ - { src: config/alertmanager/alertmanager.yml.j2, dest: config/alertmanager/alertmanager.yml }
+ - { src: config/loki/loki.yml.j2, dest: config/loki/loki.yml }
+ - { src: config/tempo/tempo.yml.j2, dest: config/tempo/tempo.yml }
+ - { src: config/alloy/config.alloy.j2, dest: config/alloy/config.alloy }
+ - { src: config/grafana/provisioning/datasources/datasources.yml.j2, dest: config/grafana/provisioning/datasources/datasources.yml }
+ when: observability_stack == 'grafana-stack'
+ notify: Restart observability stack
+
+- name: Start selected observability stack
+ community.docker.docker_compose_v2:
+ project_src: "{{ stack_root }}"
+ state: present
+ pull: missing
+ remove_orphans: true
+
+- name: Flush restart handler before health checks
+ ansible.builtin.meta: flush_handlers
+
+- name: Wait for Grafana
+ ansible.builtin.uri:
+ url: http://127.0.0.1:3000/api/health
+ status_code: 200
+ register: grafana_health
+ retries: 30
+ delay: 5
+ until: grafana_health.status == 200
+
diff --git a/ansible/roles/observability/templates/elk-grafana/config/filebeat/filebeat.yml.j2 b/ansible/roles/observability/templates/elk-grafana/config/filebeat/filebeat.yml.j2
new file mode 100644
index 0000000..7832618
--- /dev/null
+++ b/ansible/roles/observability/templates/elk-grafana/config/filebeat/filebeat.yml.j2
@@ -0,0 +1,21 @@
+filebeat.inputs:
+ - type: filestream
+ id: docker-containers
+ enabled: true
+ paths:
+ - /var/lib/docker/containers/*/*.log
+ parsers:
+ - container:
+ stream: all
+ format: docker
+
+processors:
+ - add_docker_metadata:
+ host: unix:///var/run/docker.sock
+ - add_host_metadata: ~
+
+output.logstash:
+ hosts: ["logstash:5044"]
+
+logging.level: info
+
diff --git a/ansible/roles/observability/templates/elk-grafana/config/grafana/provisioning/datasources/datasources.yml.j2 b/ansible/roles/observability/templates/elk-grafana/config/grafana/provisioning/datasources/datasources.yml.j2
new file mode 100644
index 0000000..c84399b
--- /dev/null
+++ b/ansible/roles/observability/templates/elk-grafana/config/grafana/provisioning/datasources/datasources.yml.j2
@@ -0,0 +1,18 @@
+apiVersion: 1
+
+datasources:
+ - name: Elasticsearch
+ uid: elasticsearch
+ type: elasticsearch
+ access: proxy
+ url: http://elasticsearch:9200
+ database: "logs-*"
+ isDefault: true
+ editable: true
+ jsonData:
+ index: "logs-*"
+ interval: Daily
+ timeField: "@timestamp"
+ logMessageField: message
+ logLevelField: log.level
+
diff --git a/ansible/roles/observability/templates/elk-grafana/config/logstash/pipeline/logstash.conf.j2 b/ansible/roles/observability/templates/elk-grafana/config/logstash/pipeline/logstash.conf.j2
new file mode 100644
index 0000000..80c4974
--- /dev/null
+++ b/ansible/roles/observability/templates/elk-grafana/config/logstash/pipeline/logstash.conf.j2
@@ -0,0 +1,30 @@
+input {
+ beats {
+ port => 5044
+ }
+ tcp {
+ port => 5000
+ codec => json_lines
+ }
+}
+
+filter {
+ if [message] =~ /^\s*\{/ {
+ json {
+ source => "message"
+ skip_on_invalid_json => true
+ }
+ }
+ mutate {
+ add_field => { "[@metadata][target_index]" => "logs-%{+YYYY.MM.dd}" }
+ }
+}
+
+output {
+ elasticsearch {
+ hosts => ["http://elasticsearch:9200"]
+ index => "%{[@metadata][target_index]}"
+ }
+ stdout { codec => dots }
+}
+
diff --git a/ansible/roles/observability/templates/elk-grafana/docker-compose.yml.j2 b/ansible/roles/observability/templates/elk-grafana/docker-compose.yml.j2
new file mode 100644
index 0000000..1efd7f2
--- /dev/null
+++ b/ansible/roles/observability/templates/elk-grafana/docker-compose.yml.j2
@@ -0,0 +1,86 @@
+services:
+ elasticsearch:
+ image: docker.elastic.co/elasticsearch/elasticsearch:{{ elastic_version }}
+ restart: unless-stopped
+ environment:
+ discovery.type: single-node
+ xpack.security.enabled: "false"
+ xpack.security.enrollment.enabled: "false"
+ ES_JAVA_OPTS: "-Xms{{ elasticsearch_heap }} -Xmx{{ elasticsearch_heap }}"
+ volumes:
+ - elasticsearch-data:/usr/share/elasticsearch/data
+ ports:
+ - "9200:9200"
+ healthcheck:
+ test: ["CMD-SHELL", "curl -fsS http://localhost:9200/_cluster/health || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 30
+ networks: [observability]
+
+ logstash:
+ image: docker.elastic.co/logstash/logstash:{{ elastic_version }}
+ restart: unless-stopped
+ depends_on:
+ elasticsearch:
+ condition: service_healthy
+ volumes:
+ - ./config/logstash/pipeline:/usr/share/logstash/pipeline:ro
+ - logstash-data:/usr/share/logstash/data
+ ports:
+ - "5044:5044"
+ - "5000:5000/tcp"
+ networks: [observability]
+
+ filebeat:
+ image: docker.elastic.co/beats/filebeat:{{ elastic_version }}
+ user: root
+ restart: unless-stopped
+ command: ["filebeat", "-e", "--strict.perms=false"]
+ depends_on: [logstash]
+ volumes:
+ - ./config/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
+ - /var/lib/docker/containers:/var/lib/docker/containers:ro
+ - /var/run/docker.sock:/var/run/docker.sock:ro
+ - filebeat-data:/usr/share/filebeat/data
+ networks: [observability]
+
+ kibana:
+ image: docker.elastic.co/kibana/kibana:{{ elastic_version }}
+ restart: unless-stopped
+ depends_on:
+ elasticsearch:
+ condition: service_healthy
+ environment:
+ ELASTICSEARCH_HOSTS: http://elasticsearch:9200
+ ports:
+ - "5601:5601"
+ networks: [observability]
+
+ grafana:
+ image: grafana/grafana:{{ grafana_version }}
+ restart: unless-stopped
+ depends_on:
+ elasticsearch:
+ condition: service_healthy
+ environment:
+ GF_SECURITY_ADMIN_USER: "{{ grafana_admin_user }}"
+ GF_SECURITY_ADMIN_PASSWORD: "{{ grafana_admin_password }}"
+ GF_USERS_ALLOW_SIGN_UP: "false"
+ volumes:
+ - grafana-data:/var/lib/grafana
+ - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
+ ports:
+ - "3000:3000"
+ networks: [observability]
+
+volumes:
+ elasticsearch-data:
+ logstash-data:
+ filebeat-data:
+ grafana-data:
+
+networks:
+ observability:
+ driver: bridge
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/alertmanager/alertmanager.yml.j2 b/ansible/roles/observability/templates/grafana-stack/config/alertmanager/alertmanager.yml.j2
new file mode 100644
index 0000000..d9c8578
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/alertmanager/alertmanager.yml.j2
@@ -0,0 +1,18 @@
+global:
+ resolve_timeout: 5m
+
+route:
+ receiver: default
+ group_by: [alertname, instance]
+ group_wait: 30s
+ group_interval: 5m
+ repeat_interval: 4h
+
+receivers:
+ - name: default
+{% if alertmanager_webhook_url | length > 0 %}
+ webhook_configs:
+ - url: "{{ alertmanager_webhook_url }}"
+ send_resolved: true
+{% endif %}
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/alloy/config.alloy.j2 b/ansible/roles/observability/templates/grafana-stack/config/alloy/config.alloy.j2
new file mode 100644
index 0000000..4719621
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/alloy/config.alloy.j2
@@ -0,0 +1,51 @@
+logging {
+ level = "info"
+}
+
+discovery.docker "local" {
+ host = "unix:///var/run/docker.sock"
+}
+
+discovery.relabel "docker" {
+ targets = discovery.docker.local.targets
+
+ rule {
+ source_labels = ["__meta_docker_container_name"]
+ regex = "/(.*)"
+ target_label = "container"
+ }
+}
+
+loki.source.docker "local" {
+ host = "unix:///var/run/docker.sock"
+ targets = discovery.relabel.docker.output
+ forward_to = [loki.write.default.receiver]
+}
+
+loki.write "default" {
+ endpoint {
+ url = "http://loki:3100/loki/api/v1/push"
+ }
+}
+
+otelcol.receiver.otlp "default" {
+ grpc {
+ endpoint = "0.0.0.0:4317"
+ }
+ http {
+ endpoint = "0.0.0.0:4318"
+ }
+ output {
+ traces = [otelcol.exporter.otlp.tempo.input]
+ }
+}
+
+otelcol.exporter.otlp "tempo" {
+ client {
+ endpoint = "tempo:4317"
+ tls {
+ insecure = true
+ }
+ }
+}
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/grafana/provisioning/datasources/datasources.yml.j2 b/ansible/roles/observability/templates/grafana-stack/config/grafana/provisioning/datasources/datasources.yml.j2
new file mode 100644
index 0000000..a73dc2e
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/grafana/provisioning/datasources/datasources.yml.j2
@@ -0,0 +1,54 @@
+apiVersion: 1
+
+datasources:
+ - name: Prometheus
+ uid: prometheus
+ type: prometheus
+ access: proxy
+ url: http://prometheus:9090
+ isDefault: true
+ editable: true
+ jsonData:
+ exemplarTraceIdDestinations:
+ - datasourceUid: tempo
+ name: trace_id
+
+ - name: Loki
+ uid: loki
+ type: loki
+ access: proxy
+ url: http://loki:3100
+ editable: true
+ jsonData:
+ derivedFields:
+ - datasourceUid: tempo
+ matcherRegex: 'trace[_-]?id["=: ]+([a-fA-F0-9]+)'
+ name: TraceID
+ url: '$${__value.raw}'
+
+ - name: Tempo
+ uid: tempo
+ type: tempo
+ access: proxy
+ url: http://tempo:3200
+ editable: true
+ jsonData:
+ tracesToLogsV2:
+ datasourceUid: loki
+ filterByTraceID: true
+ spanStartTimeShift: -1m
+ spanEndTimeShift: 1m
+ serviceMap:
+ datasourceUid: prometheus
+ nodeGraph:
+ enabled: true
+
+ - name: Alertmanager
+ uid: alertmanager
+ type: alertmanager
+ access: proxy
+ url: http://alertmanager:9093
+ editable: true
+ jsonData:
+ implementation: prometheus
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/loki/loki.yml.j2 b/ansible/roles/observability/templates/grafana-stack/config/loki/loki.yml.j2
new file mode 100644
index 0000000..56f5fb0
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/loki/loki.yml.j2
@@ -0,0 +1,34 @@
+auth_enabled: false
+
+server:
+ http_listen_port: 3100
+
+common:
+ path_prefix: /var/loki
+ replication_factor: 1
+ ring:
+ instance_addr: 127.0.0.1
+ kvstore:
+ store: inmemory
+
+schema_config:
+ configs:
+ - from: 2024-01-01
+ store: tsdb
+ object_store: filesystem
+ schema: v13
+ index:
+ prefix: index_
+ period: 24h
+
+storage_config:
+ filesystem:
+ directory: /var/loki/chunks
+
+compactor:
+ working_directory: /var/loki/compactor
+
+limits_config:
+ retention_period: 336h
+ allow_structured_metadata: true
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/prometheus/prometheus.yml.j2 b/ansible/roles/observability/templates/grafana-stack/config/prometheus/prometheus.yml.j2
new file mode 100644
index 0000000..a8fd10e
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/prometheus/prometheus.yml.j2
@@ -0,0 +1,43 @@
+global:
+ scrape_interval: 15s
+ evaluation_interval: 15s
+
+rule_files:
+ - /etc/prometheus/rules/*.yml
+
+alerting:
+ alertmanagers:
+ - static_configs:
+ - targets: ["alertmanager:9093"]
+
+scrape_configs:
+ - job_name: prometheus
+ static_configs:
+ - targets: ["prometheus:9090"]
+
+ - job_name: node-exporter
+ static_configs:
+ - targets: ["node-exporter:9100"]
+
+ - job_name: loki
+ static_configs:
+ - targets: ["loki:3100"]
+
+ - job_name: tempo
+ static_configs:
+ - targets: ["tempo:3200"]
+
+ - job_name: alloy
+ static_configs:
+ - targets: ["alloy:12345"]
+
+ - job_name: grafana
+ static_configs:
+ - targets: ["grafana:3000"]
+
+{% for job in prometheus_extra_scrape_jobs %}
+ - job_name: "{{ job.job_name }}"
+ static_configs:
+ - targets: {{ job.targets | to_json }}
+{% endfor %}
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/prometheus/rules/host.rules.yml.j2 b/ansible/roles/observability/templates/grafana-stack/config/prometheus/rules/host.rules.yml.j2
new file mode 100644
index 0000000..5d2ecc2
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/prometheus/rules/host.rules.yml.j2
@@ -0,0 +1,21 @@
+groups:
+ - name: host.rules
+ rules:
+ - alert: HostExporterDown
+ expr: up{job="node-exporter"} == 0
+ for: 2m
+ labels:
+ severity: critical
+ annotations:
+ summary: "Node exporter is unavailable"
+ description: "Prometheus has failed to scrape {{ '{{' }} $labels.instance {{ '}}' }} for more than two minutes."
+
+ - alert: HostHighCPU
+ expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
+ for: 10m
+ labels:
+ severity: warning
+ annotations:
+ summary: "High CPU utilization"
+ description: "CPU utilization on {{ '{{' }} $labels.instance {{ '}}' }} has exceeded 85% for ten minutes."
+
diff --git a/ansible/roles/observability/templates/grafana-stack/config/tempo/tempo.yml.j2 b/ansible/roles/observability/templates/grafana-stack/config/tempo/tempo.yml.j2
new file mode 100644
index 0000000..56d738e
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/config/tempo/tempo.yml.j2
@@ -0,0 +1,42 @@
+server:
+ http_listen_port: 3200
+
+distributor:
+ receivers:
+ otlp:
+ protocols:
+ grpc:
+ endpoint: 0.0.0.0:4317
+ http:
+ endpoint: 0.0.0.0:4318
+
+ingester:
+ max_block_duration: 5m
+
+compactor:
+ compaction:
+ block_retention: 336h
+
+storage:
+ trace:
+ backend: local
+ wal:
+ path: /var/tempo/wal
+ local:
+ path: /var/tempo/blocks
+
+metrics_generator:
+ registry:
+ external_labels:
+ source: tempo
+ storage:
+ path: /var/tempo/generator/wal
+ remote_write:
+ - url: http://prometheus:9090/api/v1/write
+ send_exemplars: true
+
+overrides:
+ defaults:
+ metrics_generator:
+ processors: [service-graphs, span-metrics, local-blocks]
+
diff --git a/ansible/roles/observability/templates/grafana-stack/docker-compose.yml.j2 b/ansible/roles/observability/templates/grafana-stack/docker-compose.yml.j2
new file mode 100644
index 0000000..38d60de
--- /dev/null
+++ b/ansible/roles/observability/templates/grafana-stack/docker-compose.yml.j2
@@ -0,0 +1,110 @@
+services:
+ prometheus:
+ image: prom/prometheus:{{ prometheus_version }}
+ restart: unless-stopped
+ command:
+ - --config.file=/etc/prometheus/prometheus.yml
+ - --storage.tsdb.path=/prometheus
+ - --storage.tsdb.retention.time=15d
+ - --web.enable-lifecycle
+ - --web.enable-remote-write-receiver
+ volumes:
+ - ./config/prometheus:/etc/prometheus:ro
+ - prometheus-data:/prometheus
+ ports:
+ - "9090:9090"
+ networks: [observability]
+
+ alertmanager:
+ image: prom/alertmanager:{{ alertmanager_version }}
+ restart: unless-stopped
+ command:
+ - --config.file=/etc/alertmanager/alertmanager.yml
+ - --storage.path=/alertmanager
+ volumes:
+ - ./config/alertmanager:/etc/alertmanager:ro
+ - alertmanager-data:/alertmanager
+ ports:
+ - "9093:9093"
+ networks: [observability]
+
+ loki:
+ image: grafana/loki:{{ loki_version }}
+ restart: unless-stopped
+ command: ["-config.file=/etc/loki/loki.yml"]
+ volumes:
+ - ./config/loki/loki.yml:/etc/loki/loki.yml:ro
+ - loki-data:/var/loki
+ ports:
+ - "3100:3100"
+ networks: [observability]
+
+ tempo:
+ image: grafana/tempo:{{ tempo_version }}
+ restart: unless-stopped
+ command: ["-config.file=/etc/tempo/tempo.yml"]
+ volumes:
+ - ./config/tempo/tempo.yml:/etc/tempo/tempo.yml:ro
+ - tempo-data:/var/tempo
+ ports:
+ - "3200:3200"
+ networks: [observability]
+
+ node-exporter:
+ image: prom/node-exporter:{{ node_exporter_version }}
+ restart: unless-stopped
+ command:
+ - --path.rootfs=/host
+ pid: host
+ volumes:
+ - /:/host:ro,rslave
+ ports:
+ - "9100:9100"
+ networks: [observability]
+
+ alloy:
+ image: grafana/alloy:{{ alloy_version }}
+ restart: unless-stopped
+ command:
+ - run
+ - --server.http.listen-addr=0.0.0.0:12345
+ - --storage.path=/var/lib/alloy/data
+ - /etc/alloy/config.alloy
+ volumes:
+ - ./config/alloy/config.alloy:/etc/alloy/config.alloy:ro
+ - /var/run/docker.sock:/var/run/docker.sock:ro
+ - alloy-data:/var/lib/alloy/data
+ ports:
+ - "12345:12345"
+ - "4317:4317"
+ - "4318:4318"
+ networks: [observability]
+
+ grafana:
+ image: grafana/grafana:{{ grafana_version }}
+ restart: unless-stopped
+ depends_on: [prometheus, loki, tempo]
+ environment:
+ GF_SECURITY_ADMIN_USER: "{{ grafana_admin_user }}"
+ GF_SECURITY_ADMIN_PASSWORD: "{{ grafana_admin_password }}"
+ GF_USERS_ALLOW_SIGN_UP: "false"
+ GF_FEATURE_TOGGLES_ENABLE: traceqlEditor
+ volumes:
+ - grafana-data:/var/lib/grafana
+ - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
+ ports:
+ - "3000:3000"
+ networks: [observability]
+
+volumes:
+ prometheus-data:
+ alertmanager-data:
+ loki-data:
+ tempo-data:
+ alloy-data:
+ grafana-data:
+
+networks:
+ observability:
+ driver: bridge
+