diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 280 |
1 files changed, 280 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..7bcea7f --- /dev/null +++ b/README.md @@ -0,0 +1,280 @@ +# telemetry + +Host telemetry collection and a Prometheus/Loki/Grafana server, as two NixOS modules you +can deploy together or apart. + +--- + +## ⚠️ This repository is vibe coded. Run it at your own risk. + +Every line here — the modules, the collectors, the dashboard generators, the recording +rules, and this README — was written by an LLM in conversation. No human has reviewed it +line by line. There is no warranty of any kind. **Read it before you deploy it**, and pay +particular attention to the systemd hardening options, the firewall rules, and anything +touching raw block devices. + +What *has* been checked: + +- every dashboard query executed against a real Prometheus fed fixture data +- `promtool` over the recording rules and the generated scrape config +- `nix flake check`: both modules evaluate, and the committed dashboard JSON matches what + the generators produce +- the collectors run against mock `smartctl`/`zpool`/`zfs`, and partially against real + hardware + +What has **not**: + +- no panel has been rendered in a real Grafana +- no output has come from a real ZFS pool +- no real SAS self-test log has been parsed + +Treat the numbers on the dashboards as unverified until you have sanity-checked them +against `zpool status` and `smartctl -x` yourself. + +--- + +## What it is + +The split is not cosmetic. The collectors read host hardware and kernel state — `/proc`, +`/sys`, SMART over raw block devices, ZFS ioctls — which a container has no view of. The +server reads none of that, so it can live wherever you like. + +| module | contains | runs where | +|---|---|---| +| `nixosModules.collectors` | node + smartctl exporters, ZFS and SMART textfile collectors, Alloy | the host, necessarily | +| `nixosModules.server` | Prometheus, Loki, Grafana, optional Caddy | anywhere that can reach the collectors | +| `nixosModules.default` | both | one machine | + +Grafana comes provisioned with four dashboards: Node Exporter Full (1860), Smartctl +Exporter (22604), and two generated here — ZFS Pools and Drive Health, a +CrystalDiskInfo-style per-drive view. + +```nix +{ + inputs.telemetry.url = "path:/home/timmy/docs/src/flakes/lgtm"; +} +``` + +The examples below take `inputs` as a module argument, which means passing it through: + +```nix +nixosConfigurations.poweredge = nixpkgs.lib.nixosSystem { + specialArgs = { inherit inputs; }; + # ... +}; +``` + +--- + +## Example: one machine + +```nix +{ inputs, ... }: +{ + imports = [ inputs.telemetry.nixosModules.default ]; + + services.telemetry = { + collectors.enable = true; + server = { + enable = true; + grafana.secretKeyFile = "/run/secrets/grafana-secret-key"; + proxy.enable = true; # Grafana on :80 + }; + }; +} +``` + +Grafana at `http://localhost`. `collectorHosts` defaults to `[ "localhost" ]`. + +## Example: web services in a container + +The host runs the collectors; the container runs everything else. This is the intended +shape. + +```nix +{ config, inputs, ... }: +{ + imports = [ inputs.telemetry.nixosModules.collectors ]; + + services.telemetry.collectors = { + enable = true; + logs.lokiUrl = "http://metrics:3100"; # the container, by name + }; + + sops.secrets.grafana-secret-key = { + sopsFile = ./secrets/grafana.yaml; + key = "secret_key"; + }; + + containers.metrics = { + autoStart = true; + privateNetwork = true; + hostBridge = "br-lan0"; + localMacAddress = "02:00:00:00:00:06"; + + # The module reads whatever path you point secretKeyFile at. + bindMounts."/run/grafana-secret-key" = { + hostPath = config.sops.secrets.grafana-secret-key.path; + isReadOnly = true; + }; + + config = { ... }: { + imports = [ inputs.telemetry.nixosModules.server ]; + + networking.enableIPv6 = false; + networking.interfaces.eth0.useDHCP = true; + + services.telemetry.server = { + enable = true; + collectorHosts = [ "poweredge" ]; # the host above + grafana.secretKeyFile = "/run/grafana-secret-key"; + proxy.enable = true; + }; + + system.stateVersion = "26.05"; # the module does not set this for you + }; + }; +} +``` + +Two names have to resolve for this to work: the container must be able to look up +`poweredge`, and the host must be able to look up `metrics`. With `hostBridge` plus DHCP +on the LAN that is your router's job; otherwise use IP addresses in both places. + +## Example: several machines + +List them, and import the collectors module on each: + +```nix +# in the container +services.telemetry.server.collectorHosts = [ "poweredge" "optiplex" ]; +``` + +```nix +# on optiplex +{ inputs, ... }: +{ + imports = [ inputs.telemetry.nixosModules.collectors ]; + + services.telemetry.collectors = { + enable = true; + logs.lokiUrl = "http://metrics:3100"; + }; +} +``` + +You get **one set of dashboards with a Host dropdown**, not a set per machine. Pick a +host and the whole page swaps to it. + +| dashboard | picker(s) | +|---|---| +| Drive Health | **Host** → SATA drive / SAS drive | +| ZFS Pools | **Host** → Pool | +| Node Exporter Full | Job → Nodename → Instance | +| Smartctl Exporter | Node | + +Dependent dropdowns repopulate on switch, so you never see poweredge's `tank` while +viewing optiplex. Hosts appear and disappear on their own — the pickers are +`label_values(..., host)`, so a machine shows up as soon as Prometheus scrapes it. + +The Host picker is **single-select with no "All"** on purpose: drive and pool names are +not unique across a fleet, and two hosts at once would make `max by (device)` silently +merge two different physical drives into one plausible-looking row. + +## Example: turning things off + +```nix +services.telemetry = { + collectors = { + enable = true; + zfs.enable = false; # no ZFS on this box + logs.enable = false; # no journal shipping + node.extraCollectors = [ "interrupts" ]; # fills 1860's IRQ panel; high cardinality + }; + + server = { + enable = true; + loki.enable = false; + grafana.dashboards.zfs-pools = false; + grafana.extraDashboards.mine = ./my-dashboard.json; + prometheus.extraScrapeConfigs = [{ + job_name = "router"; + static_configs = [{ targets = [ "router:9100" ]; }]; + }]; + }; +}; +``` + +Ports live in `services.telemetry.ports` (`node`, `smartctl`, `loki`, `prometheus`, +`grafana`) and are shared by both halves. + +Jobs added through `extraScrapeConfigs` are passed to Prometheus untouched — they get no +`host` label, so they will not show up in the dashboards' Host picker. Add your own +`relabel_configs` if you want them to. + +--- + +## Layout + +``` +flake.nix modules, collector packages, checks, dev shell +modules/ + common.nix ports, shared by both halves + collectors.nix host-side exporters and timers + server.nix Prometheus, Loki, Grafana, proxy + dashboards.nix builds dashboard JSON into a provisionable form + packages.nix the two collectors as ordinary packages +collectors/*.py the collectors themselves; runnable by hand +dashboards/gen_*.py dashboard generators +dashboards/*.json generated, committed, checked +rules/smartctl.yml Prometheus recording rules +``` + +## Working on it + +Dashboard JSON is **generated** — edit `dashboards/gen_*.py`, never the JSON: + +```sh +python3 dashboards/gen_zfs.py > dashboards/zfs-pools.json +python3 dashboards/gen_smart.py > dashboards/smart.json +``` + +`nix flake check` fails if you forget. + +The collectors are plain scripts taking `--output PATH`, so you can see exactly what a +machine reports: + +```sh +nix run .#zfs-metrics -- --output /tmp/zfs.prom +sudo nix run .#smart-selftest -- --output /tmp/smart.prom +``` + +`smart-selftest` needs privileges to open the devices. Without them smartctl still exits +0 with well-formed but empty JSON; the collector logs `no identity data` rather than +inventing placeholder values. + +## Things that will bite you + +- **`prometheus.scrapeInterval` deliberately sets two things.** Grafana derives + `$__rate_interval` from the datasource's `timeInterval`; if that disagrees with the real + scrape interval, `rate()` windows are mis-sized and rate-based panels intermittently + render "No data". Don't configure the Grafana datasource by hand. +- **The datasource UIDs `prometheus` and `loki` are fixed, not options.** The generated + dashboards reference them literally. +- **Grafana's secret key is mandatory** as of nixpkgs 26.05. `openssl rand -base64 32`, + store it somewhere, point `grafana.secretKeyFile` at it. +- **Changing `collectorHosts` changes the label set.** Panels look empty for a minute + after first deploy while the recording rules re-evaluate; data from before the change + stays under the old labels. +- **smartd does not run self-tests just because it is enabled.** NixOS's default `-a` + flags contain no `-s` schedule, so self-test history stays empty: + ```nix + services.smartd.defaults.autodetected = "-a -o on -s (S/../.././02|L/../../6/03)"; + ``` +- **SAS drives have no power-on-hours attribute.** smartctl sources `power_on_time` for + SCSI from the Background Scan Results log page, which is not the drive's lifetime meter + and often reads far too low. Drive Health takes the larger of that and the lifetime + stamp in the drive's own self-test log — so it can exceed what `smartctl -x` prints. +- **IPv6 literals need help.** The `host` label is built by stripping the port off + `__address__` with `([^:]+)(?::[0-9]+)?`, which a bracketed IPv6 address does not match. + Use `prometheus.extraScrapeConfigs` with an explicit label for those. |