# Deploying Two Observability Stacks with Ansible

This project deploys two independent stacks on a single Ubuntu/Debian host using Docker Compose:

1. **ELK + Grafana:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana
2. **Grafana Observability:** Prometheus + Alertmanager + Loki + Tempo + Grafana, along with Alloy and Node Exporter for data collection

> The default settings are suitable for Lab, Demo, and training purposes. For Production, be sure to read the "Production Notes" section.

## Operational Diagrams

- `diagrams/01-elk-grafana-operational.png` and its SVG vector version
- `diagrams/02-grafana-lgtm-operational.png` and its SVG vector version

Path colors in the second diagram:

- Green: Metrics
- Blue: Logs
- Purple: Traces
- Dashed orange: Query/Response
- Red: Alert delivery

## Project Structure

```text
inventory/hosts.yml                 SSH targets
group_vars/all.yml                  Versions, Grafana password, and variables
playbooks/elk-grafana.yml           Installs ELK + Grafana
playbooks/grafana-stack.yml         Installs Grafana/Prometheus/Loki/Tempo
roles/docker/                       Installs Docker Engine and Compose v2
roles/observability/                Templates, execution, and health checks
diagrams/                           PNG and SVG diagrams
```

## Prerequisites

- Control node: Ansible Core 2.16 or newer
- Target: Ubuntu 22.04/24.04 or Debian 12, with SSH and sudo access
- Recommended minimum for the ELK Lab: 4 vCPUs, 8 GB RAM, and 50 GB disk
- Recommended minimum for the Grafana Stack Lab: 2 vCPUs, 4 GB RAM, and 30 GB disk

On the control node:

```bash
cd observability-ansible
ansible-galaxy collection install -r requirements.yml
```

## 1. Configure the Inventory

Edit the `inventory/hosts.yml` file:

```yaml
all:
  children:
    observability:
      hosts:
        obs01:
          ansible_host: 10.10.20.15
          ansible_user: ubuntu
          ansible_ssh_private_key_file: ~/.ssh/id_ed25519
```

Test the connection:

```bash
ansible observability -m ping
```

## 2. Configure Variables and Secrets

Versions and core settings live in `group_vars/all.yml`. Be sure to change the default Grafana password. To store the password securely:

```bash
ansible-vault encrypt_string 'A-strong-password' --name grafana_admin_password
```

Replace the `grafana_admin_password` value with the Vault output, and run the playbook with `--ask-vault-pass`.

If your application exposes a `/metrics` endpoint, add it here:

```yaml
prometheus_extra_scrape_jobs:
  - job_name: checkout-api
    targets:
      - 10.20.0.15:8080
      - 10.20.0.16:8080
```

To send alerts to a webhook, also set the following value:

```yaml
alertmanager_webhook_url: "https://example.internal/alerts"
```

## 3. Install ELK + Grafana

```bash
ansible-playbook playbooks/elk-grafana.yml
```

Endpoints:

| Service | URL/Port | Purpose |
| --- | --- | --- |
| Elasticsearch | `http://SERVER:9200` | Index and Search API |
| Logstash Beats input | `SERVER:5044` | Receives Filebeat |
| Logstash JSON/TCP | `SERVER:5000` | Testing and direct JSON Lines ingest |
| Kibana | `http://SERVER:5601` | Discover and Elastic analysis |
| Grafana | `http://SERVER:3000` | Dashboards on top of Elasticsearch |

Filebeat in this project reads the Docker container logs on the same host and sends them to Logstash. Logstash parses the JSON and writes it into the daily `logs-YYYY.MM.dd` index.

Test ingest:

```bash
printf '%s\n' '{"message":"checkout completed","level":"info","service":"checkout"}' \
  | nc SERVER 5000

curl 'http://SERVER:9200/logs-*/_search?pretty'
```

## 4. Install Grafana + Prometheus + Alertmanager + Loki + Tempo

```bash
ansible-playbook playbooks/grafana-stack.yml
```

Endpoints:

| Service | URL/Port | Purpose |
| --- | --- | --- |
| Grafana | `http://SERVER:3000` | Explore, Dashboards, and correlation |
| Prometheus | `http://SERVER:9090` | Metrics, PromQL, and alert rules |
| Alertmanager | `http://SERVER:9093` | Grouping, deduplication, and routing of alerts |
| Loki | `http://SERVER:3100` | Log ingestion/query and LogQL |
| Tempo | `http://SERVER:3200` | Trace query and TraceQL |
| Alloy UI | `http://SERVER:12345` | Collector pipeline status |
| OTLP/gRPC | `SERVER:4317` | Receives traces from the application |
| OTLP/HTTP | `SERVER:4318` | Receives traces from the application |
| Node Exporter | `http://SERVER:9100/metrics` | Host-level metrics |

The Prometheus, Loki, Tempo, and Alertmanager data sources are automatically provisioned in Grafana. Grafana uses exemplars for Metrics → Trace, `trace_id` for Logs → Trace, and Prometheus for the Tempo service graph.

Example OpenTelemetry environment variables for your application:

```bash
export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://SERVER:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

Alloy pushes Docker container logs to Loki and forwards OTLP traces to Tempo. Prometheus scrapes internal services and Node Exporter. The sample rules `HostExporterDown` and `HostHighCPU` are also installed.

## Day-to-Day Operations

On the target server:

```bash
cd /opt/observability/elk-grafana
docker compose ps
docker compose logs -f --tail=100

cd /opt/observability/grafana-stack
docker compose ps
docker compose logs -f --tail=100
```

Re-running the playbook is idempotent; changing a template or version tag triggers a recreate/restart of the relevant stack. Data is kept in Docker named volumes.

## Removing a Stack

To stop and remove the containers without deleting the data:

```bash
cd /opt/observability/STACK_NAME
docker compose down
```

The `-v` flag deletes all volumes and monitoring data; only use it when you have a backup and specifically intend to do this.

## Production Notes

The default playbook intentionally builds a single-node, easy-to-understand deployment. For Production:

- Enable TLS, authentication, and RBAC on all endpoints; don't expose storage ports directly to the internet.
- Keep secrets in Ansible Vault or a Secret Manager.
- Run Elasticsearch with at least three nodes, a proper snapshot repository, and ILM.
- Use S3-compatible object storage and replication for Loki and Tempo.
- Design Prometheus with remote-write/Thanos/Mimir, retention, and adequate disk capacity.
- Run Alertmanager in HA and test your notification routes.
- Add a reverse proxy, DNS, certificates, firewall rules, and backup/restore testing.
- Upgrade versions in staging first; they are intentionally pinned in `group_vars/all.yml`.

## Official Resources

- [Elastic Docker deployment](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-docker-compose)
- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
- [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
- [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/)
- [Grafana Loki configuration](https://grafana.com/docs/loki/latest/configure/)
- [Grafana Tempo configuration](https://grafana.com/docs/tempo/latest/configuration/)
- [Grafana Alloy](https://grafana.com/docs/alloy/latest/)
- [Ansible Docker Compose v2 module](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_module.html)