LedgerLine/ansible/README.md

Deploying Two Observability Stacks with Ansible

This project deploys two independent stacks on a single Ubuntu/Debian host using Docker Compose:

1. ELK + Grafana: Filebeat → Logstash → Elasticsearch → Kibana/Grafana
2. Grafana Observability: Prometheus + Alertmanager + Loki + Tempo + Grafana, along with Alloy and Node Exporter for data collection

The default settings are suitable for Lab, Demo, and training purposes. For Production, be sure to read the "Production Notes" section.

Operational Diagrams

• diagrams/01-elk-grafana-operational.png and its SVG vector version
• diagrams/02-grafana-lgtm-operational.png and its SVG vector version

Path colors in the second diagram:

• Green: Metrics
• Blue: Logs
• Purple: Traces
• Dashed orange: Query/Response
• Red: Alert delivery

Project Structure

inventory/hosts.yml                 SSH targets
group_vars/all.yml                  Versions, Grafana password, and variables
playbooks/elk-grafana.yml           Installs ELK + Grafana
playbooks/grafana-stack.yml         Installs Grafana/Prometheus/Loki/Tempo
roles/docker/                       Installs Docker Engine and Compose v2
roles/observability/                Templates, execution, and health checks
diagrams/                           PNG and SVG diagrams

Prerequisites

• Control node: Ansible Core 2.16 or newer
• Target: Ubuntu 22.04/24.04 or Debian 12, with SSH and sudo access
• Recommended minimum for the ELK Lab: 4 vCPUs, 8 GB RAM, and 50 GB disk
• Recommended minimum for the Grafana Stack Lab: 2 vCPUs, 4 GB RAM, and 30 GB disk

On the control node:

cd observability-ansible
ansible-galaxy collection install -r requirements.yml

1. Configure the Inventory

Edit the inventory/hosts.yml file:

all:
  children:
    observability:
      hosts:
        obs01:
          ansible_host: 10.10.20.15
          ansible_user: ubuntu
          ansible_ssh_private_key_file: ~/.ssh/id_ed25519

Test the connection:

ansible observability -m ping

2. Configure Variables and Secrets

Versions and core settings live in group_vars/all.yml. Be sure to change the default Grafana password. To store the password securely:

ansible-vault encrypt_string 'A-strong-password' --name grafana_admin_password

Replace the grafana_admin_password value with the Vault output, and run the playbook with --ask-vault-pass.

If your application exposes a /metrics endpoint, add it here:

prometheus_extra_scrape_jobs:
  - job_name: checkout-api
    targets:
      - 10.20.0.15:8080
      - 10.20.0.16:8080

To send alerts to a webhook, also set the following value:

alertmanager_webhook_url: "https://example.internal/alerts"

3. Install ELK + Grafana

ansible-playbook playbooks/elk-grafana.yml

Endpoints:

Service	URL/Port	Purpose
Elasticsearch	http://SERVER:9200	Index and Search API
Logstash Beats input	SERVER:5044	Receives Filebeat
Logstash JSON/TCP	SERVER:5000	Testing and direct JSON Lines ingest
Kibana	http://SERVER:5601	Discover and Elastic analysis
Grafana	http://SERVER:3000	Dashboards on top of Elasticsearch

Filebeat in this project reads the Docker container logs on the same host and sends them to Logstash. Logstash parses the JSON and writes it into the daily logs-YYYY.MM.dd index.

Test ingest:

printf '%s\n' '{"message":"checkout completed","level":"info","service":"checkout"}' \
  | nc SERVER 5000

curl 'http://SERVER:9200/logs-*/_search?pretty'

4. Install Grafana + Prometheus + Alertmanager + Loki + Tempo

ansible-playbook playbooks/grafana-stack.yml

Endpoints:

Service	URL/Port	Purpose
Grafana	http://SERVER:3000	Explore, Dashboards, and correlation
Prometheus	http://SERVER:9090	Metrics, PromQL, and alert rules
Alertmanager	http://SERVER:9093	Grouping, deduplication, and routing of alerts
Loki	http://SERVER:3100	Log ingestion/query and LogQL
Tempo	http://SERVER:3200	Trace query and TraceQL
Alloy UI	http://SERVER:12345	Collector pipeline status
OTLP/gRPC	SERVER:4317	Receives traces from the application
OTLP/HTTP	SERVER:4318	Receives traces from the application
Node Exporter	http://SERVER:9100/metrics	Host-level metrics

The Prometheus, Loki, Tempo, and Alertmanager data sources are automatically provisioned in Grafana. Grafana uses exemplars for Metrics → Trace, trace_id for Logs → Trace, and Prometheus for the Tempo service graph.

Example OpenTelemetry environment variables for your application:

export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://SERVER:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Alloy pushes Docker container logs to Loki and forwards OTLP traces to Tempo. Prometheus scrapes internal services and Node Exporter. The sample rules HostExporterDown and HostHighCPU are also installed.

Day-to-Day Operations

On the target server:

cd /opt/observability/elk-grafana
docker compose ps
docker compose logs -f --tail=100

cd /opt/observability/grafana-stack
docker compose ps
docker compose logs -f --tail=100

Re-running the playbook is idempotent; changing a template or version tag triggers a recreate/restart of the relevant stack. Data is kept in Docker named volumes.

Removing a Stack

To stop and remove the containers without deleting the data:

cd /opt/observability/STACK_NAME
docker compose down

The -v flag deletes all volumes and monitoring data; only use it when you have a backup and specifically intend to do this.

Production Notes

The default playbook intentionally builds a single-node, easy-to-understand deployment. For Production:

• Enable TLS, authentication, and RBAC on all endpoints; don't expose storage ports directly to the internet.
• Keep secrets in Ansible Vault or a Secret Manager.
• Run Elasticsearch with at least three nodes, a proper snapshot repository, and ILM.
• Use S3-compatible object storage and replication for Loki and Tempo.
• Design Prometheus with remote-write/Thanos/Mimir, retention, and adequate disk capacity.
• Run Alertmanager in HA and test your notification routes.
• Add a reverse proxy, DNS, certificates, firewall rules, and backup/restore testing.
• Upgrade versions in staging first; they are intentionally pinned in group_vars/all.yml.

Official Resources

• Elastic Docker deployment
• Grafana provisioning
• Prometheus configuration
• Alertmanager configuration
• Grafana Loki configuration
• Grafana Tempo configuration
• Grafana Alloy
• Ansible Docker Compose v2 module