Grafana + Loki + Promtail + Node Exporter Setup with Docker Compose

[santosh]

Grafana + Loki + Promtail + Node Exporter Setup with Docker Compose

Overview
This guide covers setting up a complete monitoring and logging stack using Grafana, Loki, Promtail and Node Exporter inside Docker with Docker Compose.

Stack Components

• Grafana - Visualization and dashboards (Port 3000)
  
• Loki - Log aggregation (Port 3100)
  
• Promtail - Log collector and shipper
  
• Node Exporter - System metrics (Port 9100)
  

Step 1 - Prerequisites
Run on: Monitoring Server

Install Docker:

Code:

curl -fsSL https://get.docker.com | sh
docker --version

Install Docker Compose:

Code:

sudo apt install docker-compose -y
docker-compose --version

Step 2 - Directory Structure
Run on: Monitoring Server

Create the monitoring directory:

Code:

mkdir -p /root/24x7/monitoring/grafana-lok
cd /root/24x7/monitoring/grafana-lok

Directory structure:

Code:

grafana-lok/
├── docker-compose.yml
├── loki-config.yml
├── promtail-config.yml

Step 3 - Create Docker Compose File
Run on: Monitoring Server

Code:

nano docker-compose.yml

Add the following content:

Code:

version: '3.8'

networks:
  monitoring_network:
    driver: bridge

services:
  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - monitoring_network
    restart: unless-stopped

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - monitoring_network
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=YOUR_GRAFANA_PASSWORD
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - monitoring_network
    restart: unless-stopped

  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    ports:
      - "9100:9100"
    networks:
      - monitoring_network
    restart: unless-stopped

volumes:
  grafana_data:

Step 4 - Create Loki Configuration
Run on: Monitoring Server

Code:

nano loki-config.yml

Add the following content:

Code:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/index
    cache_location: /tmp/loki/index_cache
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 50
  ingestion_burst_size_mb: 100

compactor:
  working_directory: /tmp/loki/compactor
  shared_store: filesystem
  compaction_interval: 10m

Step 5 - Create Promtail Configuration
Run on: Monitoring Server

Code:

nano promtail-config.yml

Add the following content:

Code:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: varlogs
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/**/*.log

Step 6 - Start All Services
Run on: Monitoring Server

Start the stack:

Code:

docker-compose up -d

Verify containers are running:

Code:

docker ps

Check logs:

Code:

docker-compose logs -f grafana
docker-compose logs -f loki
docker-compose logs -f promtail
docker-compose logs -f node_exporter

Step 7 - Access Grafana

• Open browser and go to: http://YOUR_SERVER_IP:3000
  
• Login with username: admin
  
• Login with your configured password
  

Step 8 - Add Loki as Data Source

• Login to Grafana
  
• Go to Settings > Data Sources
  
• Click Add Data Source
  
• Select Loki
  
• Set URL to: http://loki:3100
  
• Click Save & Test
  

Step 9 - Add Node Exporter Metrics

• Go to Settings > Data Sources
  
• Click Add Data Source
  
• Select Prometheus
  
• Set URL to: http://node_exporter:9100
  
• Click Save & Test
  

Step 10 - Managing Services
Run on: Monitoring Server

Restart all services:

Code:

docker-compose restart

Stop all services:

Code:

docker-compose down

Port Reference

• Grafana: Port 3000
  
• Loki: Port 3100
  
• Promtail: Port 9080
  
• Node Exporter: Port 9100
  

Summary

• Grafana, Loki, Promtail and Node Exporter running inside Docker
  
• All services connected via monitoring_network bridge
  
• Loki and Promtail handle log collection and aggregation
  
• Node Exporter provides system metrics to Grafana
  
• Grafana accessible at http://YOUR_SERVER_IP:3000
  
• Always use strong passwords for Grafana admin account
  
• Consider adding Nginx reverse proxy with SSL for production