| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Turns the Alertmanager wiring sketched out in conversation into an option, and
ships the alerting rules rather than leaving them as something to copy out of a
chat log.
services.telemetry.server.alerts = {
enable = true;
to = [ "you@example.com" ];
from = "alerts@example.com";
smarthost = "smtp.fastmail.com:587";
username = "alerts@example.com";
passwordFile = config.sops.secrets.smtp-password.path;
};
passwordFile becomes smtp_auth_password_file, read at runtime, so the password
never enters the store. `settings` is merged over the generated configuration
for inhibit rules and severity routing. Alerts group by alertname and host, so
nine failing drives on one machine arrive as one mail.
rules/alerts.yml carries twelve alerts: drive failing, degraded, failed
self-test, overheating; pool not ONLINE, unrepairable data errors, over 85%
full, scrub or snapshots overdue; exporter down, collector stale, textfile parse
error. Most are one-liners because rules/smartctl.yml has already reduced drive
health to a single smartctl:health_code. ExporterDown earns its place by
covering the case where a machine vanishes and every other alert goes quiet.
Alertmanager is defined inside the existing services.prometheus block: Nix will
not let one attrset both define services.prometheus wholesale and add a path
beneath it. Same shape as the grafana settings fix in 89f2c32.
Verified by building both ways. Enabled: two promtool-checked rule files, an
amtool-checked Alertmanager config with the password file, and Prometheus
pointing at 127.0.0.1:9093. Disabled: no alertmanager unit, one rule file, no
alerting stanza. The README's rule-override and alerts.settings snippets were
also built rather than written from memory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Grafana's upstream default for security.secret_key is a constant compiled into
every release, which is why nixpkgs 26.05 stopped passing it through and started
asserting. Requiring secretKeyFile made that assertion the first thing anyone
hit, and forced a secret manager plus a container bindMount on a deployment
whose Grafana database holds no credentials at all -- both datasources are
provisioned without any.
generateSecretKey defaults to true (and to false as soon as secretKeyFile is
set), writing 32 random bytes to Grafana's data directory on first start. The
unit runs as the grafana user into a directory users.users.grafana.createHome
already owns, so there is no chown and nothing fights Grafana for the directory.
UMask=0077 makes the file 0600 on creation and an unconditional chmod repairs
one left readable by a previous version or a manual copy.
The tradeoff is stated in the option description rather than hidden: the key
lives beside the database it encrypts, so it defends against a stolen
grafana.db, not against filesystem access. secretKeyFile remains for when the
database holds a datasource password or a contact-point token.
Verified both paths build: with no secret wiring the generator unit is present
and LoadCredential points into the data directory; with secretKeyFile set the
generator is absent and LoadCredential points at the supplied path. Both emit
secret_key=$__file{...} into config.ini. The script is idempotent, leaves an
existing key untouched, and restores 0600 on a file chmodded to 0644.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
|
Replaces the single lgtm.nix with a flake exposing two NixOS modules under
services.telemetry, so the web services can move into a container declared in
the consuming config rather than one hardcoded here.
modules/collectors.nix host-side: node + smartctl exporters, the ZFS and
SMART textfile collector timers, Alloy journal
shipping. Must run on the host -- reads /proc, /sys,
raw block devices and ZFS ioctls.
modules/server.nix Prometheus, Loki, Grafana, optional Caddy.
modules/common.nix ports, shared by both halves.
modules/dashboards.nix jq fixup and grafana.com fetches.
modules/packages.nix the two collectors as ordinary packages.
Modules are exported as paths, not functions, so they evaluate with the
consumer's nixpkgs; the flake's own input only affects packages/checks/devShells.
Multi-host support. collectorHosts is a list, both scrape jobs relabel
__address__ into a `host` label, and all 133 queries in the two generated
dashboards are scoped to host=~"$host" behind a single-select Host picker.
`instance` could not serve as that key: one machine is scraped twice on
different ports (textfile metrics on :9100, smartctl_* on :9633) and the
recording rules join across that boundary.
Grafana's secret key becomes a plain secretKeyFile path wired through
LoadCredential, with an assertion -- nixpkgs 26.05 dropped the default. No
sops-nix input.
nix flake check runs promtool over the rules, diffs the committed dashboard
JSON against what the generators produce, and evaluates both modules.
Also fixes smart-selftest emitting a placeholder identity line when smartctl
cannot open a device. smartctl exits 0 with well-formed but empty JSON in that
case, so "SSD - unknown model - no serial" was reaching a dashboard row title;
it now emits nothing and lets the smartctl:ident fallback apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ
|
