split into a flake with separable collectors/server modules

Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.

  modules/collectors.nix  host-side: node + smartctl exporters, the ZFS and
                          SMART textfile collector timers, Alloy journal
                          shipping. Must run on the host -- reads /proc, /sys,
                          raw block devices and ZFS ioctls.
  modules/server.nix      Prometheus, Loki, Grafana, optional Caddy.
  modules/common.nix      ports, shared by both halves.
  modules/dashboards.nix  jq fixup and grafana.com fetches.
  modules/packages.nix    the two collectors as ordinary packages.

Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.

Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.

Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.

nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.

Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ

1 files changed, 45 insertions, 0 deletions

diff --git a/modules/packages.nix b/modules/packages.nix
new file mode 100644
index 0000000..5e76f10
--- /dev/null
+++ b/modules/packages.nix
@@ -0,0 +1,45 @@
+# The two textfile collectors, as ordinary packages.
+#
+# They are wrappers rather than inline `writeShellApplication` bodies in the module so
+# that the Python is runnable by hand -- `zfs-metrics --output /tmp/x.prom` behaves
+# exactly as the timer does, which is how every one of these metrics was validated.
+# Neither wrapper hardcodes an output path; the unit passes it.
+{ lib, writeShellApplication, python3, smartmontools, zfs }:
+
+{
+  # Pool capacity and health, vdev errors, scrub history, dataset properties and
+  # snapshot retention -- none of which node_exporter's built-in zfs collector reports,
+  # because it reads /proc/spl/kstat/zfs and that only carries ARC statistics and objset
+  # counters. OpenZFS 2.2+ also dropped the per-pool `io` kstat, so node_zfs_zpool_* is
+  # largely empty. This shells out to the zpool/zfs CLIs instead.
+  #
+  # runtimeInputs is the whole reason for the wrapper: the collector must run against
+  # the same zfs userland as the running kernel module.
+  zfs-metrics = writeShellApplication {
+    name = "zfs-metrics";
+    runtimeInputs = [ zfs ];
+    text = ''
+      exec ${python3.interpreter} ${../collectors/zfs-metrics.py} "$@"
+    '';
+    meta = {
+      description = "Export ZFS pool, dataset and snapshot metrics in Prometheus format";
+      mainProgram = "zfs-metrics";
+    };
+  };
+
+  # smartctl_exporter reads only the entry *count* out of each drive's self-test log and
+  # discards the log table, so it cannot say when a drive last ran a short or extended
+  # test or whether it passed. It also has no way to render a drive's identity as a
+  # display string, which the dashboards need for row titles. Both come from here.
+  smart-selftest = writeShellApplication {
+    name = "smart-selftest";
+    runtimeInputs = [ smartmontools ];
+    text = ''
+      exec ${python3.interpreter} ${../collectors/smart-selftest.py} "$@"
+    '';
+    meta = {
+      description = "Export SMART self-test history and drive identity in Prometheus format";
+      mainProgram = "smart-selftest";
+    };
+  };
+}