From fe770e9f386e091dbca7cf587cfe0131f10bd728 Mon Sep 17 00:00:00 2001 From: Tim Keller Date: Thu, 27 Aug 2026 23:22:24 -0500 Subject: add server.alerts: Alertmanager with email notifications Turns the Alertmanager wiring sketched out in conversation into an option, and ships the alerting rules rather than leaving them as something to copy out of a chat log. services.telemetry.server.alerts = { enable = true; to = [ "you@example.com" ]; from = "alerts@example.com"; smarthost = "smtp.fastmail.com:587"; username = "alerts@example.com"; passwordFile = config.sops.secrets.smtp-password.path; }; passwordFile becomes smtp_auth_password_file, read at runtime, so the password never enters the store. `settings` is merged over the generated configuration for inhibit rules and severity routing. Alerts group by alertname and host, so nine failing drives on one machine arrive as one mail. rules/alerts.yml carries twelve alerts: drive failing, degraded, failed self-test, overheating; pool not ONLINE, unrepairable data errors, over 85% full, scrub or snapshots overdue; exporter down, collector stale, textfile parse error. Most are one-liners because rules/smartctl.yml has already reduced drive health to a single smartctl:health_code. ExporterDown earns its place by covering the case where a machine vanishes and every other alert goes quiet. Alertmanager is defined inside the existing services.prometheus block: Nix will not let one attrset both define services.prometheus wholesale and add a path beneath it. Same shape as the grafana settings fix in 89f2c32. Verified by building both ways. Enabled: two promtool-checked rule files, an amtool-checked Alertmanager config with the password file, and Prometheus pointing at 127.0.0.1:9093. Disabled: no alertmanager unit, one rule file, no alerting stanza. The README's rule-override and alerts.settings snippets were also built rather than written from memory. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ --- README.md | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) (limited to 'README.md') diff --git a/README.md b/README.md index 5c8e1dc..f672e3a 100644 --- a/README.md +++ b/README.md @@ -202,6 +202,96 @@ Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — t --- +## Email alerts + +`alerts.enable` brings up Alertmanager, points Prometheus at it, and loads a set of +alerting rules: + +```nix +services.telemetry.server = { + enable = true; + collectorHosts = [ "poweredge" "optiplex" ]; + + alerts = { + enable = true; + to = [ "you@example.com" ]; + from = "alerts@example.com"; + smarthost = "smtp.fastmail.com:587"; + username = "alerts@example.com"; + passwordFile = config.sops.secrets.smtp-password.path; + }; +}; +``` + +`passwordFile` is handed to Alertmanager as `smtp_auth_password_file` and read at +runtime, so the password never enters the Nix store. Drop `username` and `passwordFile` +for an unauthenticated local relay. + +### What it alerts on + +Shipped in `rules/alerts.yml`, loaded unless you set `alerts.defaultRules = false`: + +| | condition | severity | +|---|---|---| +| `DriveFailing` | SMART self-assessment failed, an attribute is below its threshold, or uncorrectable errors | critical | +| `DriveCaution` | reallocated/pending sectors, or a non-empty SAS grown defect list | warning | +| `SelfTestFailed` | last short or extended self-test did not pass | critical | +| `DriveOverheating` | above 55 °C for 30 minutes | warning | +| `PoolNotOnline` | pool is DEGRADED, FAULTED, … | critical | +| `PoolDataErrors` | errors ZFS could not repair from redundancy | critical | +| `PoolNearlyFull` | over 85% | warning | +| `ScrubOverdue` | no completed scrub in 45 days | warning | +| `SnapshotsOverdue` | auto-snapshot timer has missed two intervals | warning | +| `ExporterDown` | `up == 0` for 10 minutes | critical | +| `CollectorStale` | a `.prom` file has not been refreshed in 30 minutes | warning | +| `TextfileParseError` | node\_exporter could not parse a collector's output | warning | + +Most are one-liners because `rules/smartctl.yml` has already reduced drive health to a +single `smartctl:health_code` per drive. Every alert carries the `host` label, so the +mail tells you which machine. + +`ExporterDown` matters more than it looks: without it, every other alert here goes quiet +when a machine disappears — absence of data is not absence of a problem. + +### Changing the rules + +Thresholds are not options. Copy the file, edit it, and swap it in: + +```nix +services.telemetry.server = { + alerts.defaultRules = false; + prometheus.ruleFiles = [ + "${inputs.telemetry}/rules/smartctl.yml" # keep: the health verdict lives here + ./my-alerts.yml + ]; +}; +``` + +`promtool` runs over whatever you supply at build time, so a bad expression fails +`nixos-rebuild` rather than at 3am. + +For routing — inhibit rules, a second receiver, severity-based routes — use +`alerts.settings`, which is merged over the generated Alertmanager config: + +```nix +alerts.settings.route.routes = [{ + matchers = [ "severity=\"warning\"" ]; + receiver = "email"; + repeat_interval = "7d"; # nag less about warnings +}]; +``` + +### Notes + +- Alertmanager's port is opened by `openFirewall` along with Prometheus'. Its UI is where + you silence alerts — and anyone who can reach it can silence anything, same trust model + as the Prometheus port. +- Alerts are grouped by `alertname` and `host`, so nine failing drives on one machine + arrive as one mail. `alerts.repeatInterval` (default 24h) decides how often a + still-firing alert nags you. +- Alertmanager runs `envsubst` over its config at start, so a literal `${...}` in any + string you add via `alerts.settings` will be substituted. + ## Layout ``` -- cgit v1.3.1