From fe770e9f386e091dbca7cf587cfe0131f10bd728 Mon Sep 17 00:00:00 2001 From: Tim Keller Date: Thu, 27 Aug 2026 23:22:24 -0500 Subject: add server.alerts: Alertmanager with email notifications Turns the Alertmanager wiring sketched out in conversation into an option, and ships the alerting rules rather than leaving them as something to copy out of a chat log. services.telemetry.server.alerts = { enable = true; to = [ "you@example.com" ]; from = "alerts@example.com"; smarthost = "smtp.fastmail.com:587"; username = "alerts@example.com"; passwordFile = config.sops.secrets.smtp-password.path; }; passwordFile becomes smtp_auth_password_file, read at runtime, so the password never enters the store. `settings` is merged over the generated configuration for inhibit rules and severity routing. Alerts group by alertname and host, so nine failing drives on one machine arrive as one mail. rules/alerts.yml carries twelve alerts: drive failing, degraded, failed self-test, overheating; pool not ONLINE, unrepairable data errors, over 85% full, scrub or snapshots overdue; exporter down, collector stale, textfile parse error. Most are one-liners because rules/smartctl.yml has already reduced drive health to a single smartctl:health_code. ExporterDown earns its place by covering the case where a machine vanishes and every other alert goes quiet. Alertmanager is defined inside the existing services.prometheus block: Nix will not let one attrset both define services.prometheus wholesale and add a path beneath it. Same shape as the grafana settings fix in 89f2c32. Verified by building both ways. Enabled: two promtool-checked rule files, an amtool-checked Alertmanager config with the password file, and Prometheus pointing at 127.0.0.1:9093. Disabled: no alertmanager unit, one rule file, no alerting stanza. The README's rule-override and alerts.settings snippets were also built rather than written from memory. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ --- modules/common.nix | 5 +++++ 1 file changed, 5 insertions(+) (limited to 'modules/common.nix') diff --git a/modules/common.nix b/modules/common.nix index 480aa62..fd343de 100644 --- a/modules/common.nix +++ b/modules/common.nix @@ -38,6 +38,11 @@ in Port Prometheus listens on. ''; + alertmanager = portOption 9093 '' + Port Alertmanager listens on. Only Prometheus talks to it, but the web UI is + where you silence alerts. + ''; + grafana = portOption 3000 '' Port Grafana listens on. If the reverse proxy is enabled this stays internal and the proxy is what you connect to. -- cgit v1.3.1