# telemetry Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you can deploy together or apart. --- ## ⚠️ This repository is vibe coded. Run it at your own risk. Every line here — the modules, the collectors, the dashboard generators, the recording rules, and this README — was written by an LLM in conversation. No human has reviewed it line by line. There is no warranty of any kind. **Read it before you deploy it**, and pay particular attention to the systemd hardening options, the firewall rules, and anything touching raw block devices. What *has* been checked: - every dashboard query executed against a real Prometheus fed fixture data - `promtool` over the recording rules and the generated scrape config - `nix flake check`: both modules evaluate, and the committed dashboard JSON matches what the generators produce - the collectors run against mock `smartctl`/`zpool`/`zfs`, and partially against real hardware What has **not**: - no panel has been rendered in a real Grafana - no output has come from a real ZFS pool - no real SAS self-test log has been parsed Treat the numbers on the dashboards as unverified until you have sanity-checked them against `zpool status` and `smartctl -x` yourself. --- ## What it is The split is not cosmetic. The collectors read host hardware and kernel state — `/proc`, `/sys`, SMART over raw block devices, ZFS ioctls — which a container has no view of. The server reads none of that, so it can live wherever you like. | module | contains | runs where | |---|---|---| | `nixosModules.collectors` | node + smartctl exporters, ZFS and SMART textfile collectors, Alloy | the host, necessarily | | `nixosModules.server` | Prometheus, Loki, Grafana, optional Caddy | anywhere that can reach the collectors | | `nixosModules.default` | both | one machine | Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl Exporter (22604), and two generated here — ZFS Pools and Drive Health, a CrystalDiskInfo-style per-drive view. ```nix { inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm"; } ``` The examples below take `inputs` as a module argument, which means passing it through: ```nix nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem { specialArgs = { inherit inputs; }; # ... }; ``` --- ## Example: one machine ```nix { inputs, ... }: { imports = [ inputs.telemetry.nixosModules.default ]; services.telemetry = { collectors.enable = true; server = { enable = true; grafana.secretKeyFile = "/run/secrets/grafana-secret-key"; proxy.enable = true; # Grafana on :80 }; }; } ``` Grafana at `http://localhost`. `collectorHosts` defaults to `[ "localhost" ]`. ## Example: web services in a container The host runs the collectors; the container runs everything else. This is the intended shape. ```nix { config, inputs, ... }: { imports = [ inputs.telemetry.nixosModules.collectors ]; services.telemetry.collectors = { enable = true; logs.lokiUrl = "http://metrics:3100"; # the container, by name }; sops.secrets.grafana-secret-key = { sopsFile = ./secrets/grafana.yaml; key = "secret_key"; }; containers.metrics = { autoStart = true; privateNetwork = true; hostBridge = "br-lan0"; localMacAddress = "02:00:00:00:00:06"; # The module reads whatever path you point secretKeyFile at. bindMounts."/run/grafana-secret-key" = { hostPath = config.sops.secrets.grafana-secret-key.path; isReadOnly = true; }; config = { ... }: { imports = [ inputs.telemetry.nixosModules.server ]; networking.enableIPv6 = false; networking.interfaces.eth0.useDHCP = true; services.telemetry.server = { enable = true; collectorHosts = [ "poweredge" ]; # the host above grafana.secretKeyFile = "/run/grafana-secret-key"; proxy.enable = true; }; system.stateVersion = "26.05"; # the module does not set this for you }; }; } ``` Two names have to resolve for this to work: the container must be able to look up `poweredge`, and the host must be able to look up `metrics`. With `hostBridge` plus DHCP on the LAN that is your router's job; otherwise use IP addresses in both places. ## Example: several machines List them, and import the collectors module on each: ```nix # in the container services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ]; ``` ```nix # on optiplex { inputs, ... }: { imports = [ inputs.telemetry.nixosModules.collectors ]; services.telemetry.collectors = { enable = true; logs.lokiUrl = "http://metrics:3100"; }; } ``` You get **one set of dashboards with a Host dropdown**, not a set per machine. Pick a host and the whole page swaps to it. | dashboard | picker(s) | |---|---| | Drive Health | **Host** → SATA drive / SAS drive | | ZFS Pools | **Host** → Pool | | Node Exporter Full | Job → Nodename → Instance | | Smartctl Exporter | Node | Dependent dropdowns repopulate on switch, so you never see poweredge's `tank` while viewing optiplex. Hosts appear and disappear on their own — the pickers are `label_values(..., host)`, so a machine shows up as soon as Prometheus scrapes it. The Host picker is **single-select with no "All"** on purpose: drive and pool names are not unique across a fleet, and two hosts at once would make `max by (device)` silently merge two different physical drives into one plausible-looking row. ## Example: turning things off ```nix services.telemetry = { collectors = { enable = true; zfs.enable = false; # no ZFS on this box logs.enable = false; # no journal shipping node.extraCollectors = [ "interrupts" ]; # fills 1860's IRQ panel; high cardinality }; server = { enable = true; loki.enable = false; grafana.dashboards.zfs-pools = false; grafana.extraDashboards.mine = ./my-dashboard.json; prometheus.extraScrapeConfigs = [{ job_name = "router"; static_configs = [{ targets = [ "router:9100" ]; }]; }]; }; }; ``` Ports live in `services.telemetry.ports` (`node`, `smartctl`, `loki`, `prometheus`, `grafana`) and are shared by both halves. Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — they get no `host` label, so they will not show up in the dashboards' Host picker. Add your own `relabel_configs` if you want them to. --- ## Layout ``` flake.nix modules, collector packages, checks, dev shell modules/ common.nix ports, shared by both halves collectors.nix host-side exporters and timers server.nix Prometheus, Loki, Grafana, proxy dashboards.nix builds dashboard JSON into a provisionable form packages.nix the two collectors as ordinary packages collectors/*.py the collectors themselves; runnable by hand dashboards/gen_*.py dashboard generators dashboards/*.json generated, committed, checked rules/smartctl.yml Prometheus recording rules ``` ## Working on it Dashboard JSON is **generated** — edit `dashboards/gen_*.py`, never the JSON: ```sh python3 dashboards/gen_zfs.py > dashboards/zfs-pools.json python3 dashboards/gen_smart.py > dashboards/smart.json ``` `nix flake check` fails if you forget. The collectors are plain scripts taking `--output PATH`, so you can see exactly what a machine reports: ```sh nix run .#zfs-metrics -- --output /tmp/zfs.prom sudo nix run .#smart-selftest -- --output /tmp/smart.prom ``` `smart-selftest` needs privileges to open the devices. Without them smartctl still exits 0 with well-formed but empty JSON; the collector logs `no identity data` rather than inventing placeholder values. ## Things that will bite you - **`prometheus.scrapeInterval` deliberately sets two things.** Grafana derives `$__rate_interval` from the datasource's `timeInterval`; if that disagrees with the real scrape interval, `rate()` windows are mis-sized and rate-based panels intermittently render "No data". Don't configure the Grafana datasource by hand. - **The datasource UIDs `prometheus` and `loki` are fixed, not options.** The generated dashboards reference them literally. - **Grafana's secret key is mandatory** as of nixpkgs 26.05. `openssl rand -base64 32`, store it somewhere, point `grafana.secretKeyFile` at it. - **Changing `collectorHosts` changes the label set.** Panels look empty for a minute after first deploy while the recording rules re-evaluate; data from before the change stays under the old labels. - **smartd does not run self-tests just because it is enabled.** NixOS's default `-a` flags contain no `-s` schedule, so self-test history stays empty: ```nix services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)"; ``` - **SAS drives have no power-on-hours attribute.** smartctl sources `power_on_time` for SCSI from the Background Scan Results log page, which is not the drive's lifetime meter and often reads far too low. Drive Health takes the larger of that and the lifetime stamp in the drive's own self-test log — so it can exceed what `smartctl -x` prints. - **IPv6 literals need help.** The `host` label is built by stripping the port off `__address__` with `([^:]+)(?::[0-9]+)?`, which a bracketed IPv6 address does not match. Use `prometheus.extraScrapeConfigs` with an explicit label for those.