split into a flake with separable collectors/server modules

Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.

  modules/collectors.nix  host-side: node + smartctl exporters, the ZFS and
                          SMART textfile collector timers, Alloy journal
                          shipping. Must run on the host -- reads /proc, /sys,
                          raw block devices and ZFS ioctls.
  modules/server.nix      Prometheus, Loki, Grafana, optional Caddy.
  modules/common.nix      ports, shared by both halves.
  modules/dashboards.nix  jq fixup and grafana.com fetches.
  modules/packages.nix    the two collectors as ordinary packages.

Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.

Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.

Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.

nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.

Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ

1 files changed, 15 insertions, 3 deletions

diff --git a/collectors/smart-selftest.py b/collectors/smart-selftest.py
index fa9d68f..6cb026c 100644
--- a/collectors/smart-selftest.py
+++ b/collectors/smart-selftest.py
@@ -231,13 +231,25 @@ def collect_identity(reg: Registry, name: str, data: dict) -> None:
     This exists to be read by a human in a Grafana row title, so it is deliberately a
     single rendered string rather than separate labels -- Grafana can interpolate a
     variable's display text into a title, but cannot assemble one from several labels.
+
+    Emits nothing unless smartctl actually returned identifying data. When it cannot
+    open the device -- no privileges, a controller that needs an explicit -d type -- it
+    still exits 0 with a well-formed JSON document that simply has no identity fields,
+    and filling those in with placeholders would put "unknown model · no serial" in a
+    dashboard row title. Emitting nothing instead lets the `smartctl:ident` recording
+    rule fall back to "identity pending", which is the truth.
     """
     model = (data.get("model_name") or data.get("scsi_model_name")
-             or (data.get("scsi_product") or "") or "unknown model")
-    serial = data.get("serial_number") or "no serial"
+             or data.get("scsi_product") or "")
+    serial = data.get("serial_number") or ""
     capacity = human_capacity((data.get("user_capacity") or {}).get("bytes"))
 
-    parts = [p for p in (capacity, technology(data), model, serial) if p]
+    if not (model or serial or capacity):
+        warn(f"{name}: no identity data; is smartctl able to open the device?")
+        return
+
+    parts = [p for p in (capacity, technology(data), model or "unknown model",
+                         serial or "no serial") if p]
     reg.add("smart_drive_label", 1, {"device": name, "label": " · ".join(parts)},
             "Drive identity rendered for display; the value is always 1.")