path: root/README.md

# telemetry

Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you
can deploy together or apart.

---

## ⚠️ This repository is vibe coded. Run it at your own risk.

Every line here — the modules, the collectors, the dashboard generators, the recording
rules, and this README — was written by an LLM in conversation. No human has reviewed it
line by line. There is no warranty of any kind. **Read it before you deploy it**, and pay
particular attention to the systemd hardening options, the firewall rules, and anything
touching raw block devices.

What *has* been checked:

- every dashboard query executed against a real Prometheus fed fixture data
- `promtool` over the recording rules and the generated scrape config
- `nix flake check`: both modules evaluate, and both dashboard generators run
- the collectors run against mock `smartctl`/`zpool`/`zfs`, and partially against real
  hardware

What has **not**:

- no panel has been rendered in a real Grafana
- no output has come from a real ZFS pool
- no real SAS self-test log has been parsed

Treat the numbers on the dashboards as unverified until you have sanity-checked them
against `zpool status` and `smartctl -x` yourself.

---

## What it is

The split is not cosmetic. The collectors read host hardware and kernel state — `/proc`,
`/sys`, SMART over raw block devices, ZFS ioctls — which a container has no view of. The
server reads none of that, so it can live wherever you like.

| module | contains | runs where |
|---|---|---|
| `nixosModules.collectors` | node + smartctl exporters, ZFS and SMART textfile collectors, Alloy | the host, necessarily |
| `nixosModules.server` | Prometheus, Loki, Grafana, optional Caddy | anywhere that can reach the collectors |
| `nixosModules.default` | both | one machine |

Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl
Exporter (22604), and two generated here — ZFS Pools and Drive Health, a
CrystalDiskInfo-style per-drive view.

```nix
{
  inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm";
}
```

The examples below take `inputs` as a module argument, which means passing it through:

```nix
nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem {
  specialArgs = { inherit inputs; };
  # ...
};
```

---

## Example: one machine

```nix
{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.default ];

  services.telemetry = {
    collectors.enable = true;
    server = {
      enable = true;
      grafana.secretKeyFile = "/run/secrets/grafana-secret-key";
      proxy.enable = true;                     # Grafana on :80
    };
  };
}
```

Grafana at `http://localhost`. `collectorHosts` defaults to `[ "localhost" ]`.

## Example: web services in a container

The host runs the collectors; the container runs everything else. This is the intended
shape.

```nix
{ config, inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.collectors ];

  services.telemetry.collectors = {
    enable = true;
    logs.lokiUrl = "http://metrics:3100";      # the container, by name
  };

  sops.secrets.grafana-secret-key = {
    sopsFile = ./secrets/grafana.yaml;
    key = "secret_key";
  };

  containers.metrics = {
    autoStart = true;
    privateNetwork = true;
    hostBridge = "br-lan0";
    localMacAddress = "02:00:00:00:00:06";

    # The module reads whatever path you point secretKeyFile at.
    bindMounts."/run/grafana-secret-key" = {
      hostPath = config.sops.secrets.grafana-secret-key.path;
      isReadOnly = true;
    };

    config = { ... }: {
      imports = [ inputs.telemetry.nixosModules.server ];

      networking.enableIPv6 = false;
      networking.interfaces.eth0.useDHCP = true;

      services.telemetry.server = {
        enable = true;
        collectorHosts = [ "poweredge" ];       # the host above
        grafana.secretKeyFile = "/run/grafana-secret-key";
        proxy.enable = true;
      };

      system.stateVersion = "26.05";            # the module does not set this for you
    };
  };
}
```

Two names have to resolve for this to work: the container must be able to look up
`poweredge`, and the host must be able to look up `metrics`. With `hostBridge` plus DHCP
on the LAN that is your router's job; otherwise use IP addresses in both places.

## Example: several machines

List them, and import the collectors module on each:

```nix
# in the container
services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ];
```

```nix
# on optiplex
{ inputs, ... }:
{
  imports = [ inputs.telemetry.nixosModules.collectors ];

  services.telemetry.collectors = {
    enable = true;
    logs.lokiUrl = "http://metrics:3100";
  };
}
```

You get **one set of dashboards with a Host dropdown**, not a set per machine. Pick a
host and the whole page swaps to it.

| dashboard | picker(s) |
|---|---|
| Drive Health | **Host** → SATA drive / SAS drive |
| ZFS Pools | **Host** → Pool |
| Node Exporter Full | Job → Nodename → Instance |
| Smartctl Exporter | Node |

Dependent dropdowns repopulate on switch, so you never see poweredge's `tank` while
viewing optiplex. Hosts appear and disappear on their own — the pickers are
`label_values(..., host)`, so a machine shows up as soon as Prometheus scrapes it.

The Host picker is **single-select with no "All"** on purpose: drive and pool names are
not unique across a fleet, and two hosts at once would make `max by (device)` silently
merge two different physical drives into one plausible-looking row.

## Example: turning things off

```nix
services.telemetry = {
  collectors = {
    enable = true;
    zfs.enable = false;                        # no ZFS on this box
    logs.enable = false;                       # no journal shipping
    node.extraCollectors = [ "interrupts" ];   # fills 1860's IRQ panel; high cardinality
  };

  server = {
    enable = true;
    loki.enable = false;
    grafana.dashboards.zfs-pools = false;
    grafana.extraDashboards.mine = ./my-dashboard.json;
    prometheus.extraScrapeConfigs = [{
      job_name = "router";
      static_configs = [{ targets = [ "router:9100" ]; }];
    }];
  };
};
```

Ports live in `services.telemetry.ports` (`node`, `smartctl`, `loki`, `prometheus`,
`grafana`) and are shared by both halves.

Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — they get no
`host` label, so they will not show up in the dashboards' Host picker. Add your own
`relabel_configs` if you want them to.

---

## Layout

```
flake.nix               modules, collector packages, checks, dev shell
modules/
  common.nix            ports, shared by both halves
  collectors.nix        host-side exporters and timers
  server.nix            Prometheus, Loki, Grafana, proxy
  dashboards.nix        builds dashboard JSON into a provisionable form
  packages.nix          the two collectors as ordinary packages
collectors/*.py         the collectors themselves; runnable by hand
dashboards/gen_*.py     dashboard generators; the JSON is built, not committed
rules/smartctl.yml      Prometheus recording rules
```

## Working on it

The ZFS Pools and Drive Health dashboards are **generated at build time** from
`dashboards/gen_*.py`; no JSON is committed, so there is nothing to keep in sync and no
way to ship a stale dashboard. To see what a generator currently produces:

```sh
python3 dashboards/gen_zfs.py   | less
nix build .#nixosConfigurations.<host>.config.system.build.toplevel   # or just deploy
```

(Flakes copy only git-tracked files into the store, so "gitignore the JSON and read it
from disk anyway" is not an option — it is generate, or commit.)

The collectors are plain scripts taking `--output PATH`, so you can see exactly what a
machine reports:

```sh
nix run .#zfs-metrics         -- --output /tmp/zfs.prom
sudo nix run .#smart-selftest -- --output /tmp/smart.prom
```

`smart-selftest` needs privileges to open the devices. Without them smartctl still exits
0 with well-formed but empty JSON; the collector logs `no identity data` rather than
inventing placeholder values.

## Things that will bite you

- **`prometheus.scrapeInterval` deliberately sets two things.** Grafana derives
  `$__rate_interval` from the datasource's `timeInterval`; if that disagrees with the real
  scrape interval, `rate()` windows are mis-sized and rate-based panels intermittently
  render "No data". Don't configure the Grafana datasource by hand.
- **The datasource UIDs `prometheus` and `loki` are fixed, not options.** The generated
  dashboards reference them literally.
- **Grafana's secret key is mandatory** as of nixpkgs 26.05. `openssl rand -base64 32`,
  store it somewhere, point `grafana.secretKeyFile` at it.
- **Changing `collectorHosts` changes the label set.** Panels look empty for a minute
  after first deploy while the recording rules re-evaluate; data from before the change
  stays under the old labels.
- **smartd does not run self-tests just because it is enabled.** NixOS's default `-a`
  flags contain no `-s` schedule, so self-test history stays empty:
  ```nix
  services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)";
  ```
- **SAS drives have no power-on-hours attribute.** smartctl sources `power_on_time` for
  SCSI from the Background Scan Results log page, which is not the drive's lifetime meter
  and often reads far too low. Drive Health takes the larger of that and the lifetime
  stamp in the drive's own self-test log — so it can exceed what `smartctl -x` prints.
- **IPv6 literals need help.** The `host` label is built by stripping the port off
  `__address__` with `([^:]+)(?::[0-9]+)?`, which a bracketed IPv6 address does not match.
  Use `prometheus.extraScrapeConfigs` with an explicit label for those.