| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The repeated per-drive sections are driven by $ata_device/$sas_device, which
were label_values(smartctl:row{...}, row). label_values resolves against
/api/v1/label/<l>/values over the *dashboard's* time range -- 24h by default --
so it returns every row line seen in that window, not the ones that exist now.
Replace a drive and the window still holds the line the old drive produced.
Both lines carry the same variable value, because the regex reduces them to the
bare device name that the panels select on, and Grafana's
metricNamesToVariableValues ends in uniqBy(options, 'value'), which keeps
whichever series Prometheus returned first. The dashboard then renders one row
headed with the removed drive's model and serial, no row for its replacement,
and panels underneath showing the new drive's numbers.
query_result() is the only variable form the Prometheus datasource evaluates as
an instant query (metric_find_query.ts: queryResultQuery hits /api/v1/query at
range.to), so switch both device variables to it and re-anchor the regex on the
row= label in the line Grafana renders for a vector result.
Verified against a real Prometheus loading rules/smartctl.yml, with a fixture
that replaces a SAS drive and a SATA drive in place: the old form headed sde
with OLDBAD99 and sdz with the 870 EVO it no longer contains, the new form
lists exactly the drives in the machine.
The host variable deliberately stays on label_values -- its window covering the
time range is what keeps an unreachable machine selectable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Turns the Alertmanager wiring sketched out in conversation into an option, and
ships the alerting rules rather than leaving them as something to copy out of a
chat log.
services.telemetry.server.alerts = {
enable = true;
to = [ "you@example.com" ];
from = "alerts@example.com";
smarthost = "smtp.fastmail.com:587";
username = "alerts@example.com";
passwordFile = config.sops.secrets.smtp-password.path;
};
passwordFile becomes smtp_auth_password_file, read at runtime, so the password
never enters the store. `settings` is merged over the generated configuration
for inhibit rules and severity routing. Alerts group by alertname and host, so
nine failing drives on one machine arrive as one mail.
rules/alerts.yml carries twelve alerts: drive failing, degraded, failed
self-test, overheating; pool not ONLINE, unrepairable data errors, over 85%
full, scrub or snapshots overdue; exporter down, collector stale, textfile parse
error. Most are one-liners because rules/smartctl.yml has already reduced drive
health to a single smartctl:health_code. ExporterDown earns its place by
covering the case where a machine vanishes and every other alert goes quiet.
Alertmanager is defined inside the existing services.prometheus block: Nix will
not let one attrset both define services.prometheus wholesale and add a path
beneath it. Same shape as the grafana settings fix in 89f2c32.
Verified by building both ways. Enabled: two promtool-checked rule files, an
amtool-checked Alertmanager config with the password file, and Prometheus
pointing at 127.0.0.1:9093. Disabled: no alertmanager unit, one rule file, no
alerting stanza. The README's rule-override and alerts.settings snippets were
also built rather than written from memory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Grafana's upstream default for security.secret_key is a constant compiled into
every release, which is why nixpkgs 26.05 stopped passing it through and started
asserting. Requiring secretKeyFile made that assertion the first thing anyone
hit, and forced a secret manager plus a container bindMount on a deployment
whose Grafana database holds no credentials at all -- both datasources are
provisioned without any.
generateSecretKey defaults to true (and to false as soon as secretKeyFile is
set), writing 32 random bytes to Grafana's data directory on first start. The
unit runs as the grafana user into a directory users.users.grafana.createHome
already owns, so there is no chown and nothing fights Grafana for the directory.
UMask=0077 makes the file 0600 on creation and an unconditional chmod repairs
one left readable by a previous version or a manual copy.
The tradeoff is stated in the option description rather than hidden: the key
lives beside the database it encrypts, so it defends against a stolen
grafana.db, not against filesystem access. secretKeyFile remains for when the
database holds a datasource password or a contact-point token.
Verified both paths build: with no secret wiring the generator unit is present
and LoadCredential points into the data directory; with secretKeyFile set the
generator is absent and LoadCredential points at the supplied path. Both emit
secret_key=$__file{...} into config.ini. The script is idempotent, leaves an
existing key untouched, and restores 0600 on a file chmodded to 0644.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
The JSON is machine-written and the generator beside it is the only source of
truth, so committing both only created a way for them to drift -- which is what
the dashboards-current check existed to police. modules/dashboards.nix now runs
gen_zfs.py / gen_smart.py in the derivation, so a stale dashboard cannot be
built at all and the check is gone.
Note there is no third option: flakes copy only git-tracked files into the
store, so a gitignored dashboards/*.json cannot be read from disk either. It is
generate, or commit.
Verified equivalent -- both generated dashboards are byte-identical to the JSON
this commit deletes.
Adds a dashboards-nonempty check. Factoring the shared jq invocation out first
introduced a bug it caught: the multi-line string ended in a newline, so the
callers' `< $src > "$out"` landed on a line of its own as a separate command,
truncating every dashboard to zero bytes while the build still succeeded. All
four provisioned dashboards were empty. jqFixup is now a single line, and the
check asserts each dashboard is non-empty and has a title, uid and panels.
Both failure modes are exercised: an empty dashboard and one with no panels each
fail the check. It also asserts it found four files, because the first version
globbed *.json against linkFarm entries named without the suffix and passed
vacuously.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.
modules/collectors.nix host-side: node + smartctl exporters, the ZFS and
SMART textfile collector timers, Alloy journal
shipping. Must run on the host -- reads /proc, /sys,
raw block devices and ZFS ioctls.
modules/server.nix Prometheus, Loki, Grafana, optional Caddy.
modules/common.nix ports, shared by both halves.
modules/dashboards.nix jq fixup and grafana.com fetches.
modules/packages.nix the two collectors as ordinary packages.
Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.
Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.
Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.
nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.
Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
|
