diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 90 |
1 files changed, 90 insertions, 0 deletions
@@ -202,6 +202,96 @@ Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — t --- +## Email alerts + +`alerts.enable` brings up Alertmanager, points Prometheus at it, and loads a set of +alerting rules: + +```nix +services.telemetry.server = { + enable = true; + collectorHosts = [ "poweredge" "optiplex" ]; + + alerts = { + enable = true; + to = [ "you@example.com" ]; + from = "alerts@example.com"; + smarthost = "smtp.fastmail.com:587"; + username = "alerts@example.com"; + passwordFile = config.sops.secrets.smtp-password.path; + }; +}; +``` + +`passwordFile` is handed to Alertmanager as `smtp_auth_password_file` and read at +runtime, so the password never enters the Nix store. Drop `username` and `passwordFile` +for an unauthenticated local relay. + +### What it alerts on + +Shipped in `rules/alerts.yml`, loaded unless you set `alerts.defaultRules = false`: + +| | condition | severity | +|---|---|---| +| `DriveFailing` | SMART self-assessment failed, an attribute is below its threshold, or uncorrectable errors | critical | +| `DriveCaution` | reallocated/pending sectors, or a non-empty SAS grown defect list | warning | +| `SelfTestFailed` | last short or extended self-test did not pass | critical | +| `DriveOverheating` | above 55 °C for 30 minutes | warning | +| `PoolNotOnline` | pool is DEGRADED, FAULTED, … | critical | +| `PoolDataErrors` | errors ZFS could not repair from redundancy | critical | +| `PoolNearlyFull` | over 85% | warning | +| `ScrubOverdue` | no completed scrub in 45 days | warning | +| `SnapshotsOverdue` | auto-snapshot timer has missed two intervals | warning | +| `ExporterDown` | `up == 0` for 10 minutes | critical | +| `CollectorStale` | a `.prom` file has not been refreshed in 30 minutes | warning | +| `TextfileParseError` | node\_exporter could not parse a collector's output | warning | + +Most are one-liners because `rules/smartctl.yml` has already reduced drive health to a +single `smartctl:health_code` per drive. Every alert carries the `host` label, so the +mail tells you which machine. + +`ExporterDown` matters more than it looks: without it, every other alert here goes quiet +when a machine disappears — absence of data is not absence of a problem. + +### Changing the rules + +Thresholds are not options. Copy the file, edit it, and swap it in: + +```nix +services.telemetry.server = { + alerts.defaultRules = false; + prometheus.ruleFiles = [ + "${inputs.telemetry}/rules/smartctl.yml" # keep: the health verdict lives here + ./my-alerts.yml + ]; +}; +``` + +`promtool` runs over whatever you supply at build time, so a bad expression fails +`nixos-rebuild` rather than at 3am. + +For routing — inhibit rules, a second receiver, severity-based routes — use +`alerts.settings`, which is merged over the generated Alertmanager config: + +```nix +alerts.settings.route.routes = [{ + matchers = [ "severity=\"warning\"" ]; + receiver = "email"; + repeat_interval = "7d"; # nag less about warnings +}]; +``` + +### Notes + +- Alertmanager's port is opened by `openFirewall` along with Prometheus'. Its UI is where + you silence alerts — and anyone who can reach it can silence anything, same trust model + as the Prometheus port. +- Alerts are grouped by `alertname` and `host`, so nine failing drives on one machine + arrive as one mail. `alerts.repeatInterval` (default 24h) decides how often a + still-firing alert nags you. +- Alertmanager runs `envsubst` over its config at start, so a literal `${...}` in any + string you add via `alerts.settings` will be substituted. + ## Layout ``` |