# telemetry Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you can deploy together or apart. --- ## ⚠️ This repository is vibe coded. Run it at your own risk. Every line here — the modules, the collectors, the dashboard generators, the recording rules, and this README — was written by an LLM in conversation. No human has reviewed it line by line. There is no warranty of any kind. **Read it before you deploy it**, and pay particular attention to the systemd hardening options, the firewall rules, and anything touching raw block devices. What *has* been checked: - every dashboard query executed against a real Prometheus fed fixture data - `promtool` over the recording rules and the generated scrape config - `nix flake check`: both modules evaluate, and both dashboard generators run - the collectors run against mock `smartctl`/`zpool`/`zfs`, and partially against real hardware What has **not**: - no panel has been rendered in a real Grafana - no output has come from a real ZFS pool - no real SAS self-test log has been parsed Treat the numbers on the dashboards as unverified until you have sanity-checked them against `zpool status` and `smartctl -x` yourself. --- ## What it is The split is not cosmetic. The collectors read host hardware and kernel state — `/proc`, `/sys`, SMART over raw block devices, ZFS ioctls — which a container has no view of. The server reads none of that, so it can live wherever you like. | module | contains | runs where | |---|---|---| | `nixosModules.collectors` | node + smartctl exporters, ZFS and SMART textfile collectors, Alloy | the host, necessarily | | `nixosModules.server` | Prometheus, Loki, Grafana, optional Caddy | anywhere that can reach the collectors | | `nixosModules.default` | both | one machine | Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl Exporter (22604), and two generated here — ZFS Pools and Drive Health, a CrystalDiskInfo-style per-drive view. ```nix { inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm"; } ``` The examples below take `inputs` as a module argument, which means passing it through: ```nix nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem { specialArgs = { inherit inputs; }; # ... }; ``` --- ## Example: one machine ```nix { inputs, ... }: { imports = [ inputs.telemetry.nixosModules.default ]; services.telemetry = { collectors.enable = true; server = { enable = true; proxy.enable = true; # Grafana on :80 }; }; } ``` Grafana at `http://localhost`. `collectorHosts` defaults to `[ "localhost" ]`, and Grafana's secret key is generated on first start into `/var/lib/grafana/secret-key` (mode 0600, owned by `grafana`) — see below if you would rather supply it yourself. ## Example: web services in a container The host runs the collectors; the container runs everything else. This is the intended shape. ```nix { inputs, ... }: { imports = [ inputs.telemetry.nixosModules.collectors ]; services.telemetry.collectors = { enable = true; logs.lokiUrl = "http://metrics:3100"; # the container, by name }; containers.metrics = { autoStart = true; privateNetwork = true; hostBridge = "br-lan0"; localMacAddress = "02:00:00:00:00:06"; config = { ... }: { imports = [ inputs.telemetry.nixosModules.server ]; networking.enableIPv6 = false; networking.interfaces.eth0.useDHCP = true; services.telemetry.server = { enable = true; collectorHosts = [ "poweredge" ]; # the host above proxy.enable = true; }; system.stateVersion = "26.05"; # the module does not set this for you }; }; } ``` Two names have to resolve for this to work: the container must be able to look up `poweredge`, and the host must be able to look up `metrics`. With `hostBridge` plus DHCP on the LAN that is your router's job; otherwise use IP addresses in both places. ## Example: several machines List them, and import the collectors module on each: ```nix # in the container services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ]; ``` ```nix # on optiplex { inputs, ... }: { imports = [ inputs.telemetry.nixosModules.collectors ]; services.telemetry.collectors = { enable = true; logs.lokiUrl = "http://metrics:3100"; }; } ``` You get **one set of dashboards with a Host dropdown**, not a set per machine. Pick a host and the whole page swaps to it. | dashboard | picker(s) | |---|---| | Drive Health | **Host** → SATA drive / SAS drive | | ZFS Pools | **Host** → Pool | | Node Exporter Full | Job → Nodename → Instance | | Smartctl Exporter | Node | Dependent dropdowns repopulate on switch, so you never see poweredge's `tank` while viewing optiplex. Hosts appear and disappear on their own — the pickers are `label_values(..., host)`, so a machine shows up as soon as Prometheus scrapes it. The Host picker is **single-select with no "All"** on purpose: drive and pool names are not unique across a fleet, and two hosts at once would make `max by (device)` silently merge two different physical drives into one plausible-looking row. ## Example: turning things off ```nix services.telemetry = { collectors = { enable = true; zfs.enable = false; # no ZFS on this box logs.enable = false; # no journal shipping node.extraCollectors = [ "interrupts" ]; # fills 1860's IRQ panel; high cardinality }; server = { enable = true; loki.enable = false; grafana.dashboards.zfs-pools = false; grafana.extraDashboards.mine = ./my-dashboard.json; prometheus.extraScrapeConfigs = [{ job_name = "router"; static_configs = [{ targets = [ "router:9100" ]; }]; }]; }; }; ``` Ports live in `services.telemetry.ports` (`node`, `smartctl`, `loki`, `prometheus`, `grafana`) and are shared by both halves. Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — they get no `host` label, so they will not show up in the dashboards' Host picker. Add your own `relabel_configs` if you want them to. --- ## Email alerts `alerts.enable` brings up Alertmanager, points Prometheus at it, and loads a set of alerting rules: ```nix services.telemetry.server = { enable = true; collectorHosts = [ "poweredge" "optiplex" ]; alerts = { enable = true; to = [ "you@example.com" ]; from = "alerts@example.com"; smarthost = "smtp.fastmail.com:587"; username = "alerts@example.com"; passwordFile = config.sops.secrets.smtp-password.path; }; }; ``` `passwordFile` is handed to Alertmanager as `smtp_auth_password_file` and read at runtime, so the password never enters the Nix store. Drop `username` and `passwordFile` for an unauthenticated local relay. ### What it alerts on Shipped in `rules/alerts.yml`, loaded unless you set `alerts.defaultRules = false`: | | condition | severity | |---|---|---| | `DriveFailing` | SMART self-assessment failed, an attribute is below its threshold, or uncorrectable errors | critical | | `DriveCaution` | reallocated/pending sectors, or a non-empty SAS grown defect list | warning | | `SelfTestFailed` | last short or extended self-test did not pass | critical | | `DriveOverheating` | above 55 °C for 30 minutes | warning | | `PoolNotOnline` | pool is DEGRADED, FAULTED, … | critical | | `PoolDataErrors` | errors ZFS could not repair from redundancy | critical | | `PoolNearlyFull` | over 85% | warning | | `ScrubOverdue` | no completed scrub in 45 days | warning | | `SnapshotsOverdue` | auto-snapshot timer has missed two intervals | warning | | `ExporterDown` | `up == 0` for 10 minutes | critical | | `CollectorStale` | a `.prom` file has not been refreshed in 30 minutes | warning | | `TextfileParseError` | node\_exporter could not parse a collector's output | warning | Most are one-liners because `rules/smartctl.yml` has already reduced drive health to a single `smartctl:health_code` per drive. Every alert carries the `host` label, so the mail tells you which machine. `ExporterDown` matters more than it looks: without it, every other alert here goes quiet when a machine disappears — absence of data is not absence of a problem. ### Changing the rules Thresholds are not options. Copy the file, edit it, and swap it in: ```nix services.telemetry.server = { alerts.defaultRules = false; prometheus.ruleFiles = [ "${inputs.telemetry}/rules/smartctl.yml" # keep: the health verdict lives here ./my-alerts.yml ]; }; ``` `promtool` runs over whatever you supply at build time, so a bad expression fails `nixos-rebuild` rather than at 3am. For routing — inhibit rules, a second receiver, severity-based routes — use `alerts.settings`, which is merged over the generated Alertmanager config: ```nix alerts.settings.route.routes = [{ matchers = [ "severity=\"warning\"" ]; receiver = "email"; repeat_interval = "7d"; # nag less about warnings }]; ``` ### Notes - Alertmanager's port is opened by `openFirewall` along with Prometheus'. Its UI is where you silence alerts — and anyone who can reach it can silence anything, same trust model as the Prometheus port. - Alerts are grouped by `alertname` and `host`, so nine failing drives on one machine arrive as one mail. `alerts.repeatInterval` (default 24h) decides how often a still-firing alert nags you. - Alertmanager runs `envsubst` over its config at start, so a literal `${...}` in any string you add via `alerts.settings` will be substituted. ## Layout ``` flake.nix modules, collector packages, checks, dev shell modules/ common.nix ports, shared by both halves collectors.nix host-side exporters and timers server.nix Prometheus, Loki, Grafana, proxy dashboards.nix builds dashboard JSON into a provisionable form packages.nix the two collectors as ordinary packages collectors/*.py the collectors themselves; runnable by hand dashboards/gen_*.py dashboard generators; the JSON is built, not committed rules/smartctl.yml Prometheus recording rules ``` ## Working on it The ZFS Pools and Drive Health dashboards are **generated at build time** from `dashboards/gen_*.py`; no JSON is committed, so there is nothing to keep in sync and no way to ship a stale dashboard. To see what a generator currently produces: ```sh python3 dashboards/gen_zfs.py | less nix build .#nixosConfigurations..config.system.build.toplevel # or just deploy ``` (Flakes copy only git-tracked files into the store, so "gitignore the JSON and read it from disk anyway" is not an option — it is generate, or commit.) The collectors are plain scripts taking `--output PATH`, so you can see exactly what a machine reports: ```sh nix run .#zfs-metrics -- --output /tmp/zfs.prom sudo nix run .#smart-selftest -- --output /tmp/smart.prom ``` `smart-selftest` needs privileges to open the devices. Without them smartctl still exits 0 with well-formed but empty JSON; the collector logs `no identity data` rather than inventing placeholder values. ## Things that will bite you - **`prometheus.scrapeInterval` deliberately sets two things.** Grafana derives `$__rate_interval` from the datasource's `timeInterval`; if that disagrees with the real scrape interval, `rate()` windows are mis-sized and rate-based panels intermittently render "No data". Don't configure the Grafana datasource by hand. - **The datasource UIDs `prometheus` and `loki` are fixed, not options.** The generated dashboards reference them literally. - **Grafana's secret key is mandatory** as of nixpkgs 26.05, because Grafana's upstream default is a constant compiled into every release. By default this module generates one on first start into `/var/lib/grafana/secret-key` (0600, owned by `grafana`), so there is nothing to wire up. Two things to know: it must stay *stable*, or credentials encrypted in Grafana's database stop decrypting — so keep the Grafana data directory on persistent storage; and it sits beside the database it protects, so it defends against a stolen `grafana.db` but not against filesystem access. Set `grafana.secretKeyFile` to a path you manage instead once the database holds anything worth protecting, such as a datasource password: ```nix services.telemetry.server.grafana.secretKeyFile = config.sops.secrets.grafana-secret-key.path; # turns generation off ``` - **Changing `collectorHosts` changes the label set.** Panels look empty for a minute after first deploy while the recording rules re-evaluate; data from before the change stays under the old labels. - **smartd does not run self-tests just because it is enabled.** NixOS's default `-a` flags contain no `-s` schedule, so self-test history stays empty: ```nix services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)"; ``` - **SAS drives have no power-on-hours attribute.** smartctl sources `power_on_time` for SCSI from the Background Scan Results log page, which is not the drive's lifetime meter and often reads far too low. Drive Health takes the larger of that and the lifetime stamp in the drive's own self-test log — so it can exceed what `smartctl -x` prints. - **IPv6 literals need help.** The `host` label is built by stripping the port off `__address__` with `([^:]+)(?::[0-9]+)?`, which a bracketed IPv6 address does not match. Use `prometheus.extraScrapeConfigs` with an explicit label for those.