|
Two configurations the modules claimed to support and did not.
NVMe drives had no per-drive section. smartctl:bus routes a drive by metric
presence -- ATA attribute table, or SCSI grown defect list -- and NVMe emits
neither, confirmed against a live smartctl_exporter 0.14.0 on real hardware.
With no bus label the smartctl:row join dropped them, so they appeared in Fleet
Overview and All Drives but could not be opened. Adds an nvme arm keyed on
smartctl_device_percentage_used, verdict rules over the SMART/Health
Information log, and a third repeating section.
The verdict is BAD on a critical warning, media errors, or available spare
below the drive's own threshold, and CAUTION at 90% of rated endurance.
num_err_log_entries is deliberately not in it: the NVMe drive in this machine
is healthy and sits at 2200, because it counts every command that ever
returned an error.
smart.devices reached the exporter but not the self-test collector, which
always ran its own `smartctl --scan` -- exactly what cannot see drives behind a
RAID controller, which is the only reason to set the option. The collector now
takes the same PATH[;TYPE] syntax and passes -d through.
That means it also has to reproduce the exporter's device label, which is not
the basename: /dev/bus/0 as megaraid,1 is bus_0_megaraid_1. Ported
buildDeviceLabel with its /dev/disk/by-{id,path}/ prefix stripping and the
comma rule, and the scan-time scsi -> auto fallback. Getting this wrong fails
silently -- every join in rules/smartctl.yml simply misses.
Verified: promtool; a Prometheus fixture covering all three transports plus a
megaraid-labelled device, asserting bus and verdict per drive and resolving
$ata_device/$sas_device/$nvme_device the way Grafana does; the grid layout;
the ExecStart with and without smart.devices; and the collector against this
machine's NVMe drive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.
modules/collectors.nix host-side: node + smartctl exporters, the ZFS and
SMART textfile collector timers, Alloy journal
shipping. Must run on the host -- reads /proc, /sys,
raw block devices and ZFS ioctls.
modules/server.nix Prometheus, Loki, Grafana, optional Caddy.
modules/common.nix ports, shared by both halves.
modules/dashboards.nix jq fixup and grafana.com fetches.
modules/packages.nix the two collectors as ordinary packages.
Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.
Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.
Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.
nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.
Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
