path: root/README.md

# telemetry

Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you
can deploy together or apart.

---

## ⚠️ This repository is vibe coded. Run it at your own risk.

Every line here — the modules, the collectors, the dashboard generators, the recording
rules, and this README — was written by an LLM in conversation. No human has reviewed it
line by line. There is no warranty of any kind. **Read it before you deploy it**, and pay
particular attention to the systemd hardening options, the firewall rules, and anything
touching raw block devices.

What *has* been checked:

- every dashboard query executed against a real Prometheus fed fixture data
- `promtool` over the recording rules and the generated scrape config
- `nix flake check`: both modules evaluate, and both dashboard generators run
- the collectors run against mock `smartctl`/`zpool`/`zfs`, and against a real NVMe drive
- the NVMe metric names the rules depend on, checked against a live `smartctl_exporter`

What has **not**:

- no panel has been rendered in a real Grafana
- no output has come from a real ZFS pool
- no real SAS self-test log has been parsed

Treat the numbers on the dashboards as unverified until you have sanity-checked them
against `zpool status` and `smartctl -x` yourself.

---

## What it is

The split is not cosmetic. The collectors read host hardware and kernel state — `/proc`,
`/sys`, SMART over raw block devices, ZFS ioctls — which a container has no view of. The
server reads none of that, so it can live wherever you like.

| module | contains | runs where |
|---|---|---|
| `nixosModules.collectors` | node + smartctl exporters, ZFS and SMART textfile collectors, Alloy | the host, necessarily |
| `nixosModules.server` | Prometheus, Loki, Grafana, optional Caddy | anywhere that can reach the collectors |
| `nixosModules.default` | both | one machine |

Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl
Exporter (22604), and two generated here — ZFS Pools and Drive Health, a
CrystalDiskInfo-style per-drive view.

```nix
{
  inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm";
}
```

The examples below take `inputs` as a module argument, which means passing it through:

```nix
nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem {
  specialArgs = { inherit inputs; };
  # ...
};
```

---

## Example: one machine

```nix
{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.default ];

  services.telemetry = {
    collectors.enable = true;
    server = {
      enable = true;
      proxy.enable = true;                     # Grafana on :80
    };
  };
}
```

Grafana at `http://localhost`. `collectorHosts` defaults to `[ "localhost" ]`, and
Grafana's secret key is generated on first start into `/var/lib/grafana/secret-key`
(mode 0600, owned by `grafana`) — see below if you would rather supply it yourself.

## Example: web services in a container

The host runs the collectors; the container runs everything else. This is the intended
shape.

```nix
{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.collectors ];

  services.telemetry.collectors = {
    enable = true;
    logs.lokiUrl = "http://metrics:3100";      # the container, by name
  };

  containers.metrics = {
    autoStart = true;
    privateNetwork = true;
    hostBridge = "br-lan0";
    localMacAddress = "02:00:00:00:00:06";

    config = { ... }: {
      imports = [ inputs.telemetry.nixosModules.server ];

      networking.enableIPv6 = false;
      networking.interfaces.eth0.useDHCP = true;

      services.telemetry.server = {
        enable = true;
        collectorHosts = [ "poweredge" ];       # the host above
        proxy.enable = true;
      };

      system.stateVersion = "26.05";            # the module does not set this for you
    };
  };
}
```

Two names have to resolve for this to work: the container must be able to look up
`poweredge`, and the host must be able to look up `metrics`. With `hostBridge` plus DHCP
on the LAN that is your router's job; otherwise use IP addresses in both places.

## Example: several machines

List them, and import the collectors module on each:

```nix
# in the container
services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ];
```

```nix
# on optiplex
{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.collectors ];

  services.telemetry.collectors = {
    enable = true;
    logs.lokiUrl = "http://metrics:3100";
  };
}
```

You get **one set of dashboards with a Host dropdown**, not a set per machine. Pick a
host and the whole page swaps to it.

| dashboard | picker(s) |
|---|---|
| Drive Health | **Host** → SATA drive / SAS drive / NVMe drive |
| ZFS Pools | **Host** → Pool |
| Node Exporter Full | Job → Nodename → Instance |
| Smartctl Exporter | Node |

Dependent dropdowns repopulate on switch, so you never see poweredge's `tank` while
viewing optiplex. Hosts appear and disappear on their own — the pickers are
`label_values(..., host)`, so a machine shows up as soon as Prometheus scrapes it.

The Host picker is **single-select with no "All"** on purpose: drive and pool names are
not unique across a fleet, and two hosts at once would make `max by (device)` silently
merge two different physical drives into one plausible-looking row.

## Example: turning things off

```nix
services.telemetry = {
  collectors = {
    enable = true;
    zfs.enable = false;                        # no ZFS on this box
    logs.enable = false;                       # no journal shipping
    node.extraCollectors = [ "interrupts" ];   # fills 1860's IRQ panel; high cardinality
  };

  server = {
    enable = true;
    loki.enable = false;
    grafana.dashboards.zfs-pools = false;
    grafana.extraDashboards.mine = ./my-dashboard.json;
    prometheus.extraScrapeConfigs = [{
      job_name = "router";
      static_configs = [{ targets = [ "router:9100" ]; }];
    }];
  };
};
```

Ports live in `services.telemetry.ports` (`node`, `smartctl`, `loki`, `prometheus`,
`grafana`) and are shared by both halves.

Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — they get no
`host` label, so they will not show up in the dashboards' Host picker. Add your own
`relabel_configs` if you want them to.

---

## Email alerts

`alerts.enable` brings up Alertmanager, points Prometheus at it, and loads a set of
alerting rules:

```nix
services.telemetry.server = {
  enable = true;
  collectorHosts = [ "poweredge" "optiplex" ];

  alerts = {
    enable = true;
    to        = [ "you@example.com" ];
    from      = "alerts@example.com";
    smarthost = "smtp.fastmail.com:587";
    username  = "alerts@example.com";
    passwordFile = config.sops.secrets.smtp-password.path;
  };
};
```

`passwordFile` is handed to Alertmanager as `smtp_auth_password_file` and read at
runtime, so the password never enters the Nix store. Drop `username` and `passwordFile`
for an unauthenticated local relay.

### What it alerts on

Shipped in `rules/alerts.yml`, loaded unless you set `alerts.defaultRules = false`:

| | condition | severity |
|---|---|---|
| `DriveFailing` | SMART self-assessment failed, an attribute is below its threshold, uncorrectable errors, or an NVMe critical warning | critical |
| `DriveCaution` | reallocated/pending sectors, a non-empty SAS grown defect list, or 90%+ NVMe endurance used | warning |
| `SelfTestFailed` | last short or extended self-test did not pass | critical |
| `DriveOverheating` | above 55 °C for 30 minutes | warning |
| `PoolNotOnline` | pool is DEGRADED, FAULTED, … | critical |
| `PoolDataErrors` | errors ZFS could not repair from redundancy | critical |
| `PoolNearlyFull` | over 85% | warning |
| `ScrubOverdue` | no completed scrub in 45 days | warning |
| `SnapshotsOverdue` | auto-snapshot timer has missed two intervals | warning |
| `ExporterDown` | `up == 0` for 10 minutes | critical |
| `CollectorStale` | a `.prom` file has not been refreshed in 30 minutes | warning |
| `TextfileParseError` | node\_exporter could not parse a collector's output | warning |

Most are one-liners because `rules/smartctl.yml` has already reduced drive health to a
single `smartctl:health_code` per drive. Every alert carries the `host` label, so the
mail tells you which machine.

`ExporterDown` matters more than it looks: without it, every other alert here goes quiet
when a machine disappears — absence of data is not absence of a problem.

### Changing the rules

Thresholds are not options. Copy the file, edit it, and swap it in:

```nix
services.telemetry.server = {
  alerts.defaultRules = false;
  prometheus.ruleFiles = [
    "${inputs.telemetry}/rules/smartctl.yml"   # keep: the health verdict lives here
    ./my-alerts.yml
  ];
};
```

`promtool` runs over whatever you supply at build time, so a bad expression fails
`nixos-rebuild` rather than at 3am.

For routing — inhibit rules, a second receiver, severity-based routes — use
`alerts.settings`, which is merged over the generated Alertmanager config:

```nix
alerts.settings.route.routes = [{
  matchers = [ "severity=\"warning\"" ];
  receiver = "email";
  repeat_interval = "7d";        # nag less about warnings
}];
```

### Notes

- Alertmanager's port is opened by `openFirewall` along with Prometheus'. Its UI is where
  you silence alerts — and anyone who can reach it can silence anything, same trust model
  as the Prometheus port.
- Alerts are grouped by `alertname` and `host`, so nine failing drives on one machine
  arrive as one mail. `alerts.repeatInterval` (default 24h) decides how often a
  still-firing alert nags you.
- Alertmanager runs `envsubst` over its config at start, so a literal `${...}` in any
  string you add via `alerts.settings` will be substituted.

## Layout

```
flake.nix               modules, collector packages, checks, dev shell
modules/
  common.nix            ports, shared by both halves
  collectors.nix        host-side exporters and timers
  server.nix            Prometheus, Loki, Grafana, proxy
  dashboards.nix        builds dashboard JSON into a provisionable form
  packages.nix          the two collectors as ordinary packages
collectors/*.py         the collectors themselves; runnable by hand
dashboards/gen_*.py     dashboard generators; the JSON is built, not committed
rules/smartctl.yml      Prometheus recording rules
```

## Working on it

The ZFS Pools and Drive Health dashboards are **generated at build time** from
`dashboards/gen_*.py`; no JSON is committed, so there is nothing to keep in sync and no
way to ship a stale dashboard. To see what a generator currently produces:

```sh
python3 dashboards/gen_zfs.py   | less
nix build .#nixosConfigurations.<host>.config.system.build.toplevel   # or just deploy
```

(Flakes copy only git-tracked files into the store, so "gitignore the JSON and read it
from disk anyway" is not an option — it is generate, or commit.)

The collectors are plain scripts taking `--output PATH`, so you can see exactly what a
machine reports:

```sh
nix run .#zfs-metrics         -- --output /tmp/zfs.prom
sudo nix run .#smart-selftest -- --output /tmp/smart.prom
```

`smart-selftest` needs privileges to open the devices. Without them smartctl still exits
0 with well-formed but empty JSON; the collector logs `no identity data` rather than
inventing placeholder values.

## Things that will bite you

- **`prometheus.scrapeInterval` deliberately sets two things.** Grafana derives
  `$__rate_interval` from the datasource's `timeInterval`; if that disagrees with the real
  scrape interval, `rate()` windows are mis-sized and rate-based panels intermittently
  render "No data". Don't configure the Grafana datasource by hand.
- **The datasource UIDs `prometheus` and `loki` are fixed, not options.** The generated
  dashboards reference them literally.
- **Grafana's secret key is mandatory** as of nixpkgs 26.05, because Grafana's upstream
  default is a constant compiled into every release. By default this module generates one
  on first start into `/var/lib/grafana/secret-key` (0600, owned by `grafana`), so there
  is nothing to wire up. Two things to know: it must stay *stable*, or credentials
  encrypted in Grafana's database stop decrypting — so keep the Grafana data directory on
  persistent storage; and it sits beside the database it protects, so it defends against
  a stolen `grafana.db` but not against filesystem access. Set
  `grafana.secretKeyFile` to a path you manage instead once the database holds anything
  worth protecting, such as a datasource password:
  ```nix
  services.telemetry.server.grafana.secretKeyFile =
    config.sops.secrets.grafana-secret-key.path;   # turns generation off
  ```
- **Changing `collectorHosts` changes the label set.** Panels look empty for a minute
  after first deploy while the recording rules re-evaluate; data from before the change
  stays under the old labels.
- **smartd does not run self-tests just because it is enabled.** NixOS's default `-a`
  flags contain no `-s` schedule, so self-test history stays empty:
  ```nix
  services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)";
  ```
- **SAS drives have no power-on-hours attribute.** smartctl sources `power_on_time` for
  SCSI from the Background Scan Results log page, which is not the drive's lifetime meter
  and often reads far too low. Drive Health takes the larger of that and the lifetime
  stamp in the drive's own self-test log — so it can exceed what `smartctl -x` prints.
- **Swap a drive and the identity in its row header may lag.** The header comes from
  `smart_drive_label`, written by the self-test collector on a 15 minute timer, so a
  replacement shows the removed drive's model and serial until the next run. It is only
  *stuck* if the unit is failing — the collector rewrites the whole file atomically, so a
  crash leaves the previous one in place indefinitely. Check with
  `systemctl status smart-selftest-collector.service`, and `systemctl start` it to refresh
  immediately.
- **The per-drive sections list the drives that are there *now*.** `$ata_device` and
  `$sas_device` are instant queries rather than `label_values`, so removed drives stop
  getting a row instead of lingering for the dashboard's whole time range — see the
  comment on `DEVICE_QUERY` in `dashboards/gen_smart.py`. The trade is that they need a
  sample within Prometheus' 5 minute lookback: when a machine has been unreachable longer
  than that, its sections disappear rather than rendering empty. Fleet Overview and the
  history graphs still work, because they read the range.
- **IPv6 literals need help.** The `host` label is built by stripping the port off
  `__address__` with `([^:]+)(?::[0-9]+)?`, which a bracketed IPv6 address does not match.
  Use `prometheus.extraScrapeConfigs` with an explicit label for those.