Monitoring with Grafana Loki

Below is a complete guide to set up Grafana, Loki, and Prometheus using Docker Compose, integrating data sources, and importing dashboards from Grafana Dashboards.

Note: While Grafana and Loki is very versatile it is a bit of an overkill for home server monitoring. Please check my post on beszel for a more efficient and straightforward tool

Why Monitoring

So when I first started my servers, it was pretty tough due to the electricity not being constant, server shutting down randomly and all sorts of issues which made me constantly go back and forth checking if my server is up or not and if it goes down what was the cause of it. This is why I ended trying to use a Monitoring system that would allow me to debug and at least find cause of my issues and alert me if my server where to die

Setting up Grafana, Loki and Prometheus

I decided keep all of this in one docker-compose.yml

services:
  loki:
    image: grafana/loki:latest
    container_name: loki
    volumes:
      - ./loki/config:/etc/loki
    command: -config.file=/etc/loki/config/config.yml
    ports:
      - "3100:3100"
    networks:
      - grafana-backend
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    depends_on:
      - loki
    ports:
      - "3200:3000"
    volumes:
      - ./grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=3Ze0Q!yiTZ
    restart: unless-stopped
    networks:
      - npm-proxy
      - grafana-backend

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - "9090:9090"
    networks:
      - grafana-backend
    restart: unless-stopped

networks:
  npm-proxy:
    external: true
  grafana-backend:
    external: true

Save the file as as docker-compose.yml

Setting up the Prometheus Configuration

Create a prometheus.yml file inside the ./prometheus/ directory:

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'loki'
    static_configs:
      - targets: ['loki:3100']

  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Note: This .yml file will change with the dashboard configurations. You may have to add more jobs as desired

Setting up Loki Configuration

Create a config.yml file for Loki inside the ./loki/config/ directory:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: filesystem
    schema: v13
    index:
      prefix: index_
      period: 24h

storage_config:
  filesystem:
    directory: /tmp/loki/chunks

Note: This is the basic configuration. Other configurations are available here

Start the service

Run the following commands to start the setup:

1. Build and start services:

   docker-compose up -d

2. If you encounter permission issues with Grafana:

   sudo chown -R 472:472 ./grafana

Adding Data Sources in Grafana

1. Access Grafana at http://<your-server>:3200/ (default admin password: 3Ze0Q!yiTZ).

2. Navigate to Configuration > Data Sources and click Add data source.

3. Add the following data sources:

   • Prometheus:

     • URL: http://prometheus:9090

   • Loki:

     • URL: http://loki:3100

Importing Dashboards

1. Visit Grafana Dashboards to find pre-built dashboards.

2. Download a dashboard JSON file or note the dashboard ID.

3. In Grafana, go to + > Import.

4. Enter the dashboard ID or upload the JSON file.

5. Select the relevant data sources (e.g., Prometheus or Loki) as per the dashboard requirements.

Monitor your setup

• Use the Node Exporter metrics to monitor CPU, memory, and disk usage.

• Visualize logs using the Loki data source.

• Add additional scrape configs in prometheus.yml for more services.

---

You now have a fully functional monitoring stack with Grafana, Loki, and Prometheus! Adjust configurations as needed for your environment.