Prometheus homelab monitoring stack

Server components:

• Prometheus
• Grafana
• Grafana Loki
• Prometheus
• Prometheus-snmp-exporter
• Grafana-matrix-forwarder

Configured client components:

• Promtail
• Node exporter

Todo:

• Split nixos config into several files
• Replace grafana-matrix-forwarder with something sensible
• Move secrets

Install

• Copy/symlink var_prometheus to /var/prometheus
• Create the file /var/prometheus/GRAFANASECRETS (If using matrix)
• Move prometheus.nix somewhere sensible and include it in your configuration.nix
• Enable nginx in your configuration

Forwarding alerts to matrix

Grafana prefers to push alerts using webhooks. Matrix does not have builtin-support for this. I want to move to Matrix-Webhook, but for now I am using Grafana to matrix forwarder.

Create the file /var/prometheus/GRAFANASECRETS and populate it with MATRIX_USER, MATRIX_PASSWORD and MATRIX_HOMESERVER. Note that these "secrets" will be visible in your nix store and even in your systemd service.