| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Two configurations the modules claimed to support and did not.
NVMe drives had no per-drive section. smartctl:bus routes a drive by metric
presence -- ATA attribute table, or SCSI grown defect list -- and NVMe emits
neither, confirmed against a live smartctl_exporter 0.14.0 on real hardware.
With no bus label the smartctl:row join dropped them, so they appeared in Fleet
Overview and All Drives but could not be opened. Adds an nvme arm keyed on
smartctl_device_percentage_used, verdict rules over the SMART/Health
Information log, and a third repeating section.
The verdict is BAD on a critical warning, media errors, or available spare
below the drive's own threshold, and CAUTION at 90% of rated endurance.
num_err_log_entries is deliberately not in it: the NVMe drive in this machine
is healthy and sits at 2200, because it counts every command that ever
returned an error.
smart.devices reached the exporter but not the self-test collector, which
always ran its own `smartctl --scan` -- exactly what cannot see drives behind a
RAID controller, which is the only reason to set the option. The collector now
takes the same PATH[;TYPE] syntax and passes -d through.
That means it also has to reproduce the exporter's device label, which is not
the basename: /dev/bus/0 as megaraid,1 is bus_0_megaraid_1. Ported
buildDeviceLabel with its /dev/disk/by-{id,path}/ prefix stripping and the
comma rule, and the scan-time scsi -> auto fallback. Getting this wrong fails
silently -- every join in rules/smartctl.yml simply misses.
Verified: promtool; a Prometheus fixture covering all three transports plus a
megaraid-labelled device, asserting bus and verdict per drive and resolving
$ata_device/$sas_device/$nvme_device the way Grafana does; the grid layout;
the ExecStart with and without smart.devices; and the collector against this
machine's NVMe drive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
The repeated per-drive sections are driven by $ata_device/$sas_device, which
were label_values(smartctl:row{...}, row). label_values resolves against
/api/v1/label/<l>/values over the *dashboard's* time range -- 24h by default --
so it returns every row line seen in that window, not the ones that exist now.
Replace a drive and the window still holds the line the old drive produced.
Both lines carry the same variable value, because the regex reduces them to the
bare device name that the panels select on, and Grafana's
metricNamesToVariableValues ends in uniqBy(options, 'value'), which keeps
whichever series Prometheus returned first. The dashboard then renders one row
headed with the removed drive's model and serial, no row for its replacement,
and panels underneath showing the new drive's numbers.
query_result() is the only variable form the Prometheus datasource evaluates as
an instant query (metric_find_query.ts: queryResultQuery hits /api/v1/query at
range.to), so switch both device variables to it and re-anchor the regex on the
row= label in the line Grafana renders for a vector result.
Verified against a real Prometheus loading rules/smartctl.yml, with a fixture
that replaces a SAS drive and a SATA drive in place: the old form headed sde
with OLDBAD99 and sdz with the 870 EVO it no longer contains, the new form
lists exactly the drives in the machine.
The host variable deliberately stays on label_values -- its window covering the
time range is what keeps an unreachable machine selectable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
The JSON is machine-written and the generator beside it is the only source of
truth, so committing both only created a way for them to drift -- which is what
the dashboards-current check existed to police. modules/dashboards.nix now runs
gen_zfs.py / gen_smart.py in the derivation, so a stale dashboard cannot be
built at all and the check is gone.
Note there is no third option: flakes copy only git-tracked files into the
store, so a gitignored dashboards/*.json cannot be read from disk either. It is
generate, or commit.
Verified equivalent -- both generated dashboards are byte-identical to the JSON
this commit deletes.
Adds a dashboards-nonempty check. Factoring the shared jq invocation out first
introduced a bug it caught: the multi-line string ended in a newline, so the
callers' `< $src > "$out"` landed on a line of its own as a separate command,
truncating every dashboard to zero bytes while the build still succeeded. All
four provisioned dashboards were empty. jqFixup is now a single line, and the
check asserts each dashboard is non-empty and has a title, uid and panels.
Both failure modes are exercised: an empty dashboard and one with no panels each
fail the check. It also asserts it found four files, because the first version
globbed *.json against linkFarm entries named without the suffix and passed
vacuously.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.
modules/collectors.nix host-side: node + smartctl exporters, the ZFS and
SMART textfile collector timers, Alloy journal
shipping. Must run on the host -- reads /proc, /sys,
raw block devices and ZFS ioctls.
modules/server.nix Prometheus, Loki, Grafana, optional Caddy.
modules/common.nix ports, shared by both halves.
modules/dashboards.nix jq fixup and grafana.com fetches.
modules/packages.nix the two collectors as ordinary packages.
Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.
Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.
Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.
nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.
Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
|
