telemetry

Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you can deploy together or apart.

⚠️ This repository is vibe coded. Run it at your own risk.

Every line here — the modules, the collectors, the dashboard generators, the recording rules, and this README — was written by an LLM in conversation. No human has reviewed it line by line. There is no warranty of any kind. Read it before you deploy it, and pay particular attention to the systemd hardening options, the firewall rules, and anything touching raw block devices.

What has been checked:

• every dashboard query executed against a real Prometheus fed fixture data
• promtool over the recording rules and the generated scrape config
• nix flake check: both modules evaluate, and both dashboard generators run
• the collectors run against mock smartctl/zpool/zfs, and against a real NVMe drive
• the NVMe metric names the rules depend on, checked against a live smartctl_exporter

What has not:

• no panel has been rendered in a real Grafana
• no output has come from a real ZFS pool
• no real SAS self-test log has been parsed

Treat the numbers on the dashboards as unverified until you have sanity-checked them against zpool status and smartctl -x yourself.

What it is

The split is not cosmetic. The collectors read host hardware and kernel state — /proc, /sys, SMART over raw block devices, ZFS ioctls — which a container has no view of. The server reads none of that, so it can live wherever you like.

Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl Exporter (22604), and two generated here — ZFS Pools and Drive Health, a CrystalDiskInfo-style per-drive view.

{
  inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm";
}

The examples below take inputs as a module argument, which means passing it through:

nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem {
  specialArgs = { inherit inputs; };
  # ...
};

Example: one machine

{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.default ];

  services.telemetry = {
    collectors.enable = true;
    server = {
      enable = true;
      proxy.enable = true;                     # Grafana on :80
    };
  };
}

Grafana at http://localhost. collectorHosts defaults to [ "localhost" ], and Grafana's secret key is generated on first start into /var/lib/grafana/secret-key (mode 0600, owned by grafana) — see below if you would rather supply it yourself.

Example: web services in a container

The host runs the collectors; the container runs everything else. This is the intended shape.

{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.collectors ];

  services.telemetry.collectors = {
    enable = true;
    logs.lokiUrl = "http://metrics:3100";      # the container, by name
  };

  containers.metrics = {
    autoStart = true;
    privateNetwork = true;
    hostBridge = "br-lan0";
    localMacAddress = "02:00:00:00:00:06";

    config = { ... }: {
      imports = [ inputs.telemetry.nixosModules.server ];

      networking.enableIPv6 = false;
      networking.interfaces.eth0.useDHCP = true;

      services.telemetry.server = {
        enable = true;
        collectorHosts = [ "poweredge" ];       # the host above
        proxy.enable = true;
      };

      system.stateVersion = "26.05";            # the module does not set this for you
    };
  };
}

Two names have to resolve for this to work: the container must be able to look up poweredge, and the host must be able to look up metrics. With hostBridge plus DHCP on the LAN that is your router's job; otherwise use IP addresses in both places.

Example: several machines

List them, and import the collectors module on each:

# in the container
services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ];

# on optiplex
{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.collectors ];

  services.telemetry.collectors = {
    enable = true;
    logs.lokiUrl = "http://metrics:3100";
  };
}

You get one set of dashboards with a Host dropdown, not a set per machine. Pick a host and the whole page swaps to it.

Dependent dropdowns repopulate on switch, so you never see poweredge's tank while viewing optiplex. Hosts appear and disappear on their own — the pickers are label_values(..., host), so a machine shows up as soon as Prometheus scrapes it.

The Host picker is single-select with no "All" on purpose: drive and pool names are not unique across a fleet, and two hosts at once would make max by (device) silently merge two different physical drives into one plausible-looking row.

Example: turning things off

services.telemetry = {
  collectors = {
    enable = true;
    zfs.enable = false;                        # no ZFS on this box
    logs.enable = false;                       # no journal shipping
    node.extraCollectors = [ "interrupts" ];   # fills 1860's IRQ panel; high cardinality
  };

  server = {
    enable = true;
    loki.enable = false;
    grafana.dashboards.zfs-pools = false;
    grafana.extraDashboards.mine = ./my-dashboard.json;
    prometheus.extraScrapeConfigs = [{
      job_name = "router";
      static_configs = [{ targets = [ "router:9100" ]; }];
    }];
  };
};

Ports live in services.telemetry.ports (node, smartctl, loki, prometheus, grafana) and are shared by both halves.

Jobs added through extraScrapeConfigs are passed to Prometheus untouched — they get no host label, so they will not show up in the dashboards' Host picker. Add your own relabel_configs if you want them to.

Email alerts

alerts.enable brings up Alertmanager, points Prometheus at it, and loads a set of alerting rules:

services.telemetry.server = {
  enable = true;
  collectorHosts = [ "poweredge" "optiplex" ];

  alerts = {
    enable = true;
    to        = [ "you@example.com" ];
    from      = "alerts@example.com";
    smarthost = "smtp.fastmail.com:587";
    username  = "alerts@example.com";
    passwordFile = config.sops.secrets.smtp-password.path;
  };
};

passwordFile is handed to Alertmanager as smtp_auth_password_file and read at runtime, so the password never enters the Nix store. Drop username and passwordFile for an unauthenticated local relay.

What it alerts on

Shipped in rules/alerts.yml, loaded unless you set alerts.defaultRules = false:

Most are one-liners because rules/smartctl.yml has already reduced drive health to a single smartctl:health_code per drive. Every alert carries the host label, so the mail tells you which machine.

ExporterDown matters more than it looks: without it, every other alert here goes quiet when a machine disappears — absence of data is not absence of a problem.

Changing the rules

Thresholds are not options. Copy the file, edit it, and swap it in:

services.telemetry.server = {
  alerts.defaultRules = false;
  prometheus.ruleFiles = [
    "${inputs.telemetry}/rules/smartctl.yml"   # keep: the health verdict lives here
    ./my-alerts.yml
  ];
};

promtool runs over whatever you supply at build time, so a bad expression fails nixos-rebuild rather than at 3am.

For routing — inhibit rules, a second receiver, severity-based routes — use alerts.settings, which is merged over the generated Alertmanager config:

alerts.settings.route.routes = [{
  matchers = [ "severity=\"warning\"" ];
  receiver = "email";
  repeat_interval = "7d";        # nag less about warnings
}];

Notes

• Alertmanager's port is opened by openFirewall along with Prometheus'. Its UI is where you silence alerts — and anyone who can reach it can silence anything, same trust model as the Prometheus port.
• Alerts are grouped by alertname and host, so nine failing drives on one machine arrive as one mail. alerts.repeatInterval (default 24h) decides how often a still-firing alert nags you.
• Alertmanager runs envsubst over its config at start, so a literal ${...} in any string you add via alerts.settings will be substituted.

Layout

flake.nix               modules, collector packages, checks, dev shell
modules/
  common.nix            ports, shared by both halves
  collectors.nix        host-side exporters and timers
  server.nix            Prometheus, Loki, Grafana, proxy
  dashboards.nix        builds dashboard JSON into a provisionable form
  packages.nix          the two collectors as ordinary packages
collectors/*.py         the collectors themselves; runnable by hand
dashboards/gen_*.py     dashboard generators; the JSON is built, not committed
rules/smartctl.yml      Prometheus recording rules

Working on it

The ZFS Pools and Drive Health dashboards are generated at build time from dashboards/gen_*.py; no JSON is committed, so there is nothing to keep in sync and no way to ship a stale dashboard. To see what a generator currently produces:

python3 dashboards/gen_zfs.py   | less
nix build .#nixosConfigurations.<host>.config.system.build.toplevel   # or just deploy

(Flakes copy only git-tracked files into the store, so "gitignore the JSON and read it from disk anyway" is not an option — it is generate, or commit.)

The collectors are plain scripts taking --output PATH, so you can see exactly what a machine reports:

nix run .#zfs-metrics         -- --output /tmp/zfs.prom
sudo nix run .#smart-selftest -- --output /tmp/smart.prom

smart-selftest needs privileges to open the devices. Without them smartctl still exits 0 with well-formed but empty JSON; the collector logs no identity data rather than inventing placeholder values.

Things that will bite you

• prometheus.scrapeInterval deliberately sets two things. Grafana derives $__rate_interval from the datasource's timeInterval; if that disagrees with the real scrape interval, rate() windows are mis-sized and rate-based panels intermittently render "No data". Don't configure the Grafana datasource by hand.
• The datasource UIDs prometheus and loki are fixed, not options. The generated dashboards reference them literally.
• Grafana's secret key is mandatory as of nixpkgs 26.05, because Grafana's upstream default is a constant compiled into every release. By default this module generates one on first start into /var/lib/grafana/secret-key (0600, owned by grafana), so there is nothing to wire up. Two things to know: it must stay stable, or credentials encrypted in Grafana's database stop decrypting — so keep the Grafana data directory on persistent storage; and it sits beside the database it protects, so it defends against a stolen grafana.db but not against filesystem access. Set grafana.secretKeyFile to a path you manage instead once the database holds anything worth protecting, such as a datasource password: nix services.telemetry.server.grafana.secretKeyFile = config.sops.secrets.grafana-secret-key.path; # turns generation off
• Changing collectorHosts changes the label set. Panels look empty for a minute after first deploy while the recording rules re-evaluate; data from before the change stays under the old labels.
• smartd does not run self-tests just because it is enabled. NixOS's default -a flags contain no -s schedule, so self-test history stays empty: nix services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)";
• SAS drives have no power-on-hours attribute. smartctl sources power_on_time for SCSI from the Background Scan Results log page, which is not the drive's lifetime meter and often reads far too low. Drive Health takes the larger of that and the lifetime stamp in the drive's own self-test log — so it can exceed what smartctl -x prints.
• Swap a drive and the identity in its row header may lag. The header comes from smart_drive_label, written by the self-test collector on a 15 minute timer, so a replacement shows the removed drive's model and serial until the next run. It is only stuck if the unit is failing — the collector rewrites the whole file atomically, so a crash leaves the previous one in place indefinitely. Check with systemctl status smart-selftest-collector.service, and systemctl start it to refresh immediately.
• The per-drive sections list the drives that are there now. $ata_device and $sas_device are instant queries rather than label_values, so removed drives stop getting a row instead of lingering for the dashboard's whole time range — see the comment on DEVICE_QUERY in dashboards/gen_smart.py. The trade is that they need a sample within Prometheus' 5 minute lookback: when a machine has been unreachable longer than that, its sections disappear rather than rendering empty. Fleet Overview and the history graphs still work, because they read the range.
• IPv6 literals need help. The host label is built by stripping the port off __address__ with ([^:]+)(?::[0-9]+)?, which a bracketed IPv6 address does not match. Use prometheus.extraScrapeConfigs with an explicit label for those.