Add observability platform Ansible project
Deploys the Grafana stack (Prometheus/Alertmanager/Loki/Tempo/Grafana with Alloy as collector) and the ELK stack (Filebeat/Logstash/Elasticsearch/ Kibana) via Docker Compose on a target host. External to the application repo, pre-built by the mentor.
This commit is contained in:
Movses Aslanyan
@@ -0,0 +1,204 @@
|
||||
# Deploying Two Observability Stacks with Ansible
|
||||
|
||||
This project deploys two independent stacks on a single Ubuntu/Debian host using Docker Compose:
|
||||
|
||||
1. **ELK + Grafana:** Filebeat → Logstash → Elasticsearch → Kibana/Grafana
|
||||
2. **Grafana Observability:** Prometheus + Alertmanager + Loki + Tempo + Grafana, along with Alloy and Node Exporter for data collection
|
||||
|
||||
> The default settings are suitable for Lab, Demo, and training purposes. For Production, be sure to read the "Production Notes" section.
|
||||
|
||||
## Operational Diagrams
|
||||
|
||||
- `diagrams/01-elk-grafana-operational.png` and its SVG vector version
|
||||
- `diagrams/02-grafana-lgtm-operational.png` and its SVG vector version
|
||||
|
||||
Path colors in the second diagram:
|
||||
|
||||
- Green: Metrics
|
||||
- Blue: Logs
|
||||
- Purple: Traces
|
||||
- Dashed orange: Query/Response
|
||||
- Red: Alert delivery
|
||||
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
inventory/hosts.yml SSH targets
|
||||
group_vars/all.yml Versions, Grafana password, and variables
|
||||
playbooks/elk-grafana.yml Installs ELK + Grafana
|
||||
playbooks/grafana-stack.yml Installs Grafana/Prometheus/Loki/Tempo
|
||||
roles/docker/ Installs Docker Engine and Compose v2
|
||||
roles/observability/ Templates, execution, and health checks
|
||||
diagrams/ PNG and SVG diagrams
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Control node: Ansible Core 2.16 or newer
|
||||
- Target: Ubuntu 22.04/24.04 or Debian 12, with SSH and sudo access
|
||||
- Recommended minimum for the ELK Lab: 4 vCPUs, 8 GB RAM, and 50 GB disk
|
||||
- Recommended minimum for the Grafana Stack Lab: 2 vCPUs, 4 GB RAM, and 30 GB disk
|
||||
|
||||
On the control node:
|
||||
|
||||
```bash
|
||||
cd observability-ansible
|
||||
ansible-galaxy collection install -r requirements.yml
|
||||
```
|
||||
|
||||
## 1. Configure the Inventory
|
||||
|
||||
Edit the `inventory/hosts.yml` file:
|
||||
|
||||
```yaml
|
||||
all:
|
||||
children:
|
||||
observability:
|
||||
hosts:
|
||||
obs01:
|
||||
ansible_host: 10.10.20.15
|
||||
ansible_user: ubuntu
|
||||
ansible_ssh_private_key_file: ~/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
Test the connection:
|
||||
|
||||
```bash
|
||||
ansible observability -m ping
|
||||
```
|
||||
|
||||
## 2. Configure Variables and Secrets
|
||||
|
||||
Versions and core settings live in `group_vars/all.yml`. Be sure to change the default Grafana password. To store the password securely:
|
||||
|
||||
```bash
|
||||
ansible-vault encrypt_string 'A-strong-password' --name grafana_admin_password
|
||||
```
|
||||
|
||||
Replace the `grafana_admin_password` value with the Vault output, and run the playbook with `--ask-vault-pass`.
|
||||
|
||||
If your application exposes a `/metrics` endpoint, add it here:
|
||||
|
||||
```yaml
|
||||
prometheus_extra_scrape_jobs:
|
||||
- job_name: checkout-api
|
||||
targets:
|
||||
- 10.20.0.15:8080
|
||||
- 10.20.0.16:8080
|
||||
```
|
||||
|
||||
To send alerts to a webhook, also set the following value:
|
||||
|
||||
```yaml
|
||||
alertmanager_webhook_url: "https://example.internal/alerts"
|
||||
```
|
||||
|
||||
## 3. Install ELK + Grafana
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/elk-grafana.yml
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
|
||||
| Service | URL/Port | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Elasticsearch | `http://SERVER:9200` | Index and Search API |
|
||||
| Logstash Beats input | `SERVER:5044` | Receives Filebeat |
|
||||
| Logstash JSON/TCP | `SERVER:5000` | Testing and direct JSON Lines ingest |
|
||||
| Kibana | `http://SERVER:5601` | Discover and Elastic analysis |
|
||||
| Grafana | `http://SERVER:3000` | Dashboards on top of Elasticsearch |
|
||||
|
||||
Filebeat in this project reads the Docker container logs on the same host and sends them to Logstash. Logstash parses the JSON and writes it into the daily `logs-YYYY.MM.dd` index.
|
||||
|
||||
Test ingest:
|
||||
|
||||
```bash
|
||||
printf '%s\n' '{"message":"checkout completed","level":"info","service":"checkout"}' \
|
||||
| nc SERVER 5000
|
||||
|
||||
curl 'http://SERVER:9200/logs-*/_search?pretty'
|
||||
```
|
||||
|
||||
## 4. Install Grafana + Prometheus + Alertmanager + Loki + Tempo
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/grafana-stack.yml
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
|
||||
| Service | URL/Port | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Grafana | `http://SERVER:3000` | Explore, Dashboards, and correlation |
|
||||
| Prometheus | `http://SERVER:9090` | Metrics, PromQL, and alert rules |
|
||||
| Alertmanager | `http://SERVER:9093` | Grouping, deduplication, and routing of alerts |
|
||||
| Loki | `http://SERVER:3100` | Log ingestion/query and LogQL |
|
||||
| Tempo | `http://SERVER:3200` | Trace query and TraceQL |
|
||||
| Alloy UI | `http://SERVER:12345` | Collector pipeline status |
|
||||
| OTLP/gRPC | `SERVER:4317` | Receives traces from the application |
|
||||
| OTLP/HTTP | `SERVER:4318` | Receives traces from the application |
|
||||
| Node Exporter | `http://SERVER:9100/metrics` | Host-level metrics |
|
||||
|
||||
The Prometheus, Loki, Tempo, and Alertmanager data sources are automatically provisioned in Grafana. Grafana uses exemplars for Metrics → Trace, `trace_id` for Logs → Trace, and Prometheus for the Tempo service graph.
|
||||
|
||||
Example OpenTelemetry environment variables for your application:
|
||||
|
||||
```bash
|
||||
export OTEL_SERVICE_NAME=checkout-api
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://SERVER:4318
|
||||
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
```
|
||||
|
||||
Alloy pushes Docker container logs to Loki and forwards OTLP traces to Tempo. Prometheus scrapes internal services and Node Exporter. The sample rules `HostExporterDown` and `HostHighCPU` are also installed.
|
||||
|
||||
## Day-to-Day Operations
|
||||
|
||||
On the target server:
|
||||
|
||||
```bash
|
||||
cd /opt/observability/elk-grafana
|
||||
docker compose ps
|
||||
docker compose logs -f --tail=100
|
||||
|
||||
cd /opt/observability/grafana-stack
|
||||
docker compose ps
|
||||
docker compose logs -f --tail=100
|
||||
```
|
||||
|
||||
Re-running the playbook is idempotent; changing a template or version tag triggers a recreate/restart of the relevant stack. Data is kept in Docker named volumes.
|
||||
|
||||
## Removing a Stack
|
||||
|
||||
To stop and remove the containers without deleting the data:
|
||||
|
||||
```bash
|
||||
cd /opt/observability/STACK_NAME
|
||||
docker compose down
|
||||
```
|
||||
|
||||
The `-v` flag deletes all volumes and monitoring data; only use it when you have a backup and specifically intend to do this.
|
||||
|
||||
## Production Notes
|
||||
|
||||
The default playbook intentionally builds a single-node, easy-to-understand deployment. For Production:
|
||||
|
||||
- Enable TLS, authentication, and RBAC on all endpoints; don't expose storage ports directly to the internet.
|
||||
- Keep secrets in Ansible Vault or a Secret Manager.
|
||||
- Run Elasticsearch with at least three nodes, a proper snapshot repository, and ILM.
|
||||
- Use S3-compatible object storage and replication for Loki and Tempo.
|
||||
- Design Prometheus with remote-write/Thanos/Mimir, retention, and adequate disk capacity.
|
||||
- Run Alertmanager in HA and test your notification routes.
|
||||
- Add a reverse proxy, DNS, certificates, firewall rules, and backup/restore testing.
|
||||
- Upgrade versions in staging first; they are intentionally pinned in `group_vars/all.yml`.
|
||||
|
||||
## Official Resources
|
||||
|
||||
- [Elastic Docker deployment](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-docker-compose)
|
||||
- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
|
||||
- [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
|
||||
- [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/)
|
||||
- [Grafana Loki configuration](https://grafana.com/docs/loki/latest/configure/)
|
||||
- [Grafana Tempo configuration](https://grafana.com/docs/tempo/latest/configuration/)
|
||||
- [Grafana Alloy](https://grafana.com/docs/alloy/latest/)
|
||||
- [Ansible Docker Compose v2 module](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_module.html)
|
||||
Reference in New Issue
Block a user