split into a flake with separable collectors/server modules

Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.

  modules/collectors.nix  host-side: node + smartctl exporters, the ZFS and
                          SMART textfile collector timers, Alloy journal
                          shipping. Must run on the host -- reads /proc, /sys,
                          raw block devices and ZFS ioctls.
  modules/server.nix      Prometheus, Loki, Grafana, optional Caddy.
  modules/common.nix      ports, shared by both halves.
  modules/dashboards.nix  jq fixup and grafana.com fetches.
  modules/packages.nix    the two collectors as ordinary packages.

Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.

Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.

Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.

nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.

Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ

1 files changed, 75 insertions, 0 deletions

diff --git a/modules/dashboards.nix b/modules/dashboards.nix
new file mode 100644
index 0000000..125e3e2
--- /dev/null
+++ b/modules/dashboards.nix
@@ -0,0 +1,75 @@
+# Grafana dashboards, built into a form file-provisioning can actually use.
+#
+# Returns an attrset of { name = derivation; }, where each derivation is a single
+# dashboard JSON file. modules/server.nix drops these into /etc/grafana-dashboards.
+{ lib, runCommand, fetchurl, jq, dsUid }:
+
+let
+  # Dashboards exported from grafana.com carry an `__inputs` block and reference their
+  # datasource as "''${DS_PROMETHEUS}". That placeholder is only substituted by Grafana's
+  # interactive *import* flow -- file provisioning never runs it, so the uid stays
+  # literal and every panel fails to resolve a datasource. Substitute at build time.
+  #
+  # A dashboard-level `datasource`-type template variable (lowercase, e.g.
+  # "''${ds_prometheus}") is a different mechanism and resolves fine on its own; this
+  # only touches names declared in __inputs.
+  fixupProgram = ''
+    def dsuid($pluginId):
+      if   $pluginId == "prometheus" then $prom
+      elif $pluginId == "loki"       then $loki
+      else "" end;
+
+    def subst($name; $uid):
+      walk(if type == "string" then gsub("\\$\\{" + $name + "\\}"; $uid) else . end);
+
+    reduce (.__inputs // [])[] as $i (.; subst($i.name; dsuid($i.pluginId)))
+    | del(.__inputs, .__requires)
+    # Provisioned dashboards must not carry an id, or Grafana rejects them on collision
+    # with whatever already occupies that id in its database.
+    | .id = null
+  '';
+
+  fixup = name: src: runCommand "grafana-dashboard-${name}.json"
+    { nativeBuildInputs = [ jq ]; }
+    ''
+      jq --arg prom ${lib.escapeShellArg dsUid.prometheus} \
+         --arg loki ${lib.escapeShellArg dsUid.loki} \
+         ${lib.escapeShellArg fixupProgram} < ${src} > "$out"
+    '';
+
+  fromGrafanaCom = { name, id, rev, hash }:
+    fixup name (fetchurl {
+      url = "https://grafana.com/api/dashboards/${toString id}/revisions/${toString rev}/download";
+      inherit hash;
+    });
+in
+{
+  # https://grafana.com/grafana/dashboards/1860
+  node-exporter-full = fromGrafanaCom {
+    name = "node-exporter-full";
+    id = 1860;
+    rev = 45;
+    hash = "sha256-GExrdAnzBtp1Ul13cvcZRbEM6iOtFrXXjEaY6g6lGYY=";
+  };
+
+  # https://grafana.com/grafana/dashboards/22604
+  smartctl-exporter = fromGrafanaCom {
+    name = "smartctl-exporter";
+    id = 22604;
+    rev = 2;
+    hash = "sha256-ci8WE23fZ+ltEKFoUdNNVXsUIV0jqtas79ia2lYIo88=";
+  };
+
+  # Dashboard 15362 ("ZFS Pool Metrics") used to fill this slot. It is an *InfluxDB*
+  # dashboard -- its queries are InfluxQL against a `zpool_stats` measurement -- so it
+  # can never render against Prometheus, however it is provisioned. This is a
+  # Prometheus-native replacement built on the metrics we actually export: node
+  # exporter's zfs collector plus the ZFS textfile collector.
+  # Generated by dashboards/gen_zfs.py.
+  zfs-pools = fixup "zfs-pools" ../dashboards/zfs-pools.json;
+
+  # CrystalDiskInfo-style per-drive SMART view. Generated by dashboards/gen_smart.py.
+  # The Good/Caution/Bad verdict and the per-drive row titles both come from the
+  # recording rules in rules/smartctl.yml, so this one needs those loaded to be useful.
+  drive-health = fixup "drive-health" ../dashboards/smart.json;
+}