split into a flake with separable collectors/server modules

Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.

  modules/collectors.nix  host-side: node + smartctl exporters, the ZFS and
                          SMART textfile collector timers, Alloy journal
                          shipping. Must run on the host -- reads /proc, /sys,
                          raw block devices and ZFS ioctls.
  modules/server.nix      Prometheus, Loki, Grafana, optional Caddy.
  modules/common.nix      ports, shared by both halves.
  modules/dashboards.nix  jq fixup and grafana.com fetches.
  modules/packages.nix    the two collectors as ordinary packages.

Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.

Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.

Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.

nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.

Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ

1 files changed, 280 insertions, 0 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..7bcea7f
--- /dev/null
+++ b/README.md
@@ -0,0 +1,280 @@
+# telemetry
+
+Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you
+can deploy together or apart.
+
+---
+
+## ⚠️ This repository is vibe coded. Run it at your own risk.
+
+Every line here — the modules, the collectors, the dashboard generators, the recording
+rules, and this README — was written by an LLM in conversation. No human has reviewed it
+line by line. There is no warranty of any kind. **Read it before you deploy it**, and pay
+particular attention to the systemd hardening options, the firewall rules, and anything
+touching raw block devices.
+
+What *has* been checked:
+
+- every dashboard query executed against a real Prometheus fed fixture data
+- `promtool` over the recording rules and the generated scrape config
+- `nix flake check`: both modules evaluate, and the committed dashboard JSON matches what
+  the generators produce
+- the collectors run against mock `smartctl`/`zpool`/`zfs`, and partially against real
+  hardware
+
+What has **not**:
+
+- no panel has been rendered in a real Grafana
+- no output has come from a real ZFS pool
+- no real SAS self-test log has been parsed
+
+Treat the numbers on the dashboards as unverified until you have sanity-checked them
+against `zpool status` and `smartctl -x` yourself.
+
+---
+
+## What it is
+
+The split is not cosmetic. The collectors read host hardware and kernel state — `/proc`,
+`/sys`, SMART over raw block devices, ZFS ioctls — which a container has no view of. The
+server reads none of that, so it can live wherever you like.
+
+| module | contains | runs where |
+|---|---|---|
+| `nixosModules.collectors` | node + smartctl exporters, ZFS and SMART textfile collectors, Alloy | the host, necessarily |
+| `nixosModules.server` | Prometheus, Loki, Grafana, optional Caddy | anywhere that can reach the collectors |
+| `nixosModules.default` | both | one machine |
+
+Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl
+Exporter (22604), and two generated here — ZFS Pools and Drive Health, a
+CrystalDiskInfo-style per-drive view.
+
+```nix
+{
+  inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm";
+}
+```
+
+The examples below take `inputs` as a module argument, which means passing it through:
+
+```nix
+nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem {
+  specialArgs = { inherit inputs; };
+  # ...
+};
+```
+
+---
+
+## Example: one machine
+
+```nix
+{ inputs, ... }:
+{
+  imports = [ inputs.telemetry.nixosModules.default ];
+
+  services.telemetry = {
+    collectors.enable = true;
+    server = {
+      enable = true;
+      grafana.secretKeyFile = "/run/secrets/grafana-secret-key";
+      proxy.enable = true;                     # Grafana on :80
+    };
+  };
+}
+```
+
+Grafana at `http://localhost`. `collectorHosts` defaults to `[ "localhost" ]`.
+
+## Example: web services in a container
+
+The host runs the collectors; the container runs everything else. This is the intended
+shape.
+
+```nix
+{ config, inputs, ... }:
+{
+  imports = [ inputs.telemetry.nixosModules.collectors ];
+
+  services.telemetry.collectors = {
+    enable = true;
+    logs.lokiUrl = "http://metrics:3100";      # the container, by name
+  };
+
+  sops.secrets.grafana-secret-key = {
+    sopsFile = ./secrets/grafana.yaml;
+    key = "secret_key";
+  };
+
+  containers.metrics = {
+    autoStart = true;
+    privateNetwork = true;
+    hostBridge = "br-lan0";
+    localMacAddress = "02:00:00:00:00:06";
+
+    # The module reads whatever path you point secretKeyFile at.
+    bindMounts."/run/grafana-secret-key" = {
+      hostPath = config.sops.secrets.grafana-secret-key.path;
+      isReadOnly = true;
+    };
+
+    config = { ... }: {
+      imports = [ inputs.telemetry.nixosModules.server ];
+
+      networking.enableIPv6 = false;
+      networking.interfaces.eth0.useDHCP = true;
+
+      services.telemetry.server = {
+        enable = true;
+        collectorHosts = [ "poweredge" ];       # the host above
+        grafana.secretKeyFile = "/run/grafana-secret-key";
+        proxy.enable = true;
+      };
+
+      system.stateVersion = "26.05";            # the module does not set this for you
+    };
+  };
+}
+```
+
+Two names have to resolve for this to work: the container must be able to look up
+`poweredge`, and the host must be able to look up `metrics`. With `hostBridge` plus DHCP
+on the LAN that is your router's job; otherwise use IP addresses in both places.
+
+## Example: several machines
+
+List them, and import the collectors module on each:
+
+```nix
+# in the container
+services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ];
+```
+
+```nix
+# on optiplex
+{ inputs, ... }:
+{
+  imports = [ inputs.telemetry.nixosModules.collectors ];
+
+  services.telemetry.collectors = {
+    enable = true;
+    logs.lokiUrl = "http://metrics:3100";
+  };
+}
+```
+
+You get **one set of dashboards with a Host dropdown**, not a set per machine. Pick a
+host and the whole page swaps to it.
+
+| dashboard | picker(s) |
+|---|---|
+| Drive Health | **Host** → SATA drive / SAS drive |
+| ZFS Pools | **Host** → Pool |
+| Node Exporter Full | Job → Nodename → Instance |
+| Smartctl Exporter | Node |
+
+Dependent dropdowns repopulate on switch, so you never see poweredge's `tank` while
+viewing optiplex. Hosts appear and disappear on their own — the pickers are
+`label_values(..., host)`, so a machine shows up as soon as Prometheus scrapes it.
+
+The Host picker is **single-select with no "All"** on purpose: drive and pool names are
+not unique across a fleet, and two hosts at once would make `max by (device)` silently
+merge two different physical drives into one plausible-looking row.
+
+## Example: turning things off
+
+```nix
+services.telemetry = {
+  collectors = {
+    enable = true;
+    zfs.enable = false;                        # no ZFS on this box
+    logs.enable = false;                       # no journal shipping
+    node.extraCollectors = [ "interrupts" ];   # fills 1860's IRQ panel; high cardinality
+  };
+
+  server = {
+    enable = true;
+    loki.enable = false;
+    grafana.dashboards.zfs-pools = false;
+    grafana.extraDashboards.mine = ./my-dashboard.json;
+    prometheus.extraScrapeConfigs = [{
+      job_name = "router";
+      static_configs = [{ targets = [ "router:9100" ]; }];
+    }];
+  };
+};
+```
+
+Ports live in `services.telemetry.ports` (`node`, `smartctl`, `loki`, `prometheus`,
+`grafana`) and are shared by both halves.
+
+Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — they get no
+`host` label, so they will not show up in the dashboards' Host picker. Add your own
+`relabel_configs` if you want them to.
+
+---
+
+## Layout
+
+```
+flake.nix               modules, collector packages, checks, dev shell
+modules/
+  common.nix            ports, shared by both halves
+  collectors.nix        host-side exporters and timers
+  server.nix            Prometheus, Loki, Grafana, proxy
+  dashboards.nix        builds dashboard JSON into a provisionable form
+  packages.nix          the two collectors as ordinary packages
+collectors/*.py         the collectors themselves; runnable by hand
+dashboards/gen_*.py     dashboard generators
+dashboards/*.json       generated, committed, checked
+rules/smartctl.yml      Prometheus recording rules
+```
+
+## Working on it
+
+Dashboard JSON is **generated** — edit `dashboards/gen_*.py`, never the JSON:
+
+```sh
+python3 dashboards/gen_zfs.py   > dashboards/zfs-pools.json
+python3 dashboards/gen_smart.py > dashboards/smart.json
+```
+
+`nix flake check` fails if you forget.
+
+The collectors are plain scripts taking `--output PATH`, so you can see exactly what a
+machine reports:
+
+```sh
+nix run .#zfs-metrics         -- --output /tmp/zfs.prom
+sudo nix run .#smart-selftest -- --output /tmp/smart.prom
+```
+
+`smart-selftest` needs privileges to open the devices. Without them smartctl still exits
+0 with well-formed but empty JSON; the collector logs `no identity data` rather than
+inventing placeholder values.
+
+## Things that will bite you
+
+- **`prometheus.scrapeInterval` deliberately sets two things.** Grafana derives
+  `$__rate_interval` from the datasource's `timeInterval`; if that disagrees with the real
+  scrape interval, `rate()` windows are mis-sized and rate-based panels intermittently
+  render "No data". Don't configure the Grafana datasource by hand.
+- **The datasource UIDs `prometheus` and `loki` are fixed, not options.** The generated
+  dashboards reference them literally.
+- **Grafana's secret key is mandatory** as of nixpkgs 26.05. `openssl rand -base64 32`,
+  store it somewhere, point `grafana.secretKeyFile` at it.
+- **Changing `collectorHosts` changes the label set.** Panels look empty for a minute
+  after first deploy while the recording rules re-evaluate; data from before the change
+  stays under the old labels.
+- **smartd does not run self-tests just because it is enabled.** NixOS's default `-a`
+  flags contain no `-s` schedule, so self-test history stays empty:
+  ```nix
+  services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)";
+  ```
+- **SAS drives have no power-on-hours attribute.** smartctl sources `power_on_time` for
+  SCSI from the Background Scan Results log page, which is not the drive's lifetime meter
+  and often reads far too low. Drive Health takes the larger of that and the lifetime
+  stamp in the drive's own self-test log — so it can exceed what `smartctl -x` prints.
+- **IPv6 literals need help.** The `host` label is built by stripping the port off
+  `__address__` with `([^:]+)(?::[0-9]+)?`, which a bracketed IPv6 address does not match.
+  Use `prometheus.extraScrapeConfigs` with an explicit label for those.