From e88a880435ebdda6e50bf16a493abb85b4e80d15 Mon Sep 17 00:00:00 2001 From: Tim Keller Date: Wed, 2 Sep 2026 21:01:46 -0500 Subject: add NVMe support, and give the self-test collector the device list Two configurations the modules claimed to support and did not. NVMe drives had no per-drive section. smartctl:bus routes a drive by metric presence -- ATA attribute table, or SCSI grown defect list -- and NVMe emits neither, confirmed against a live smartctl_exporter 0.14.0 on real hardware. With no bus label the smartctl:row join dropped them, so they appeared in Fleet Overview and All Drives but could not be opened. Adds an nvme arm keyed on smartctl_device_percentage_used, verdict rules over the SMART/Health Information log, and a third repeating section. The verdict is BAD on a critical warning, media errors, or available spare below the drive's own threshold, and CAUTION at 90% of rated endurance. num_err_log_entries is deliberately not in it: the NVMe drive in this machine is healthy and sits at 2200, because it counts every command that ever returned an error. smart.devices reached the exporter but not the self-test collector, which always ran its own `smartctl --scan` -- exactly what cannot see drives behind a RAID controller, which is the only reason to set the option. The collector now takes the same PATH[;TYPE] syntax and passes -d through. That means it also has to reproduce the exporter's device label, which is not the basename: /dev/bus/0 as megaraid,1 is bus_0_megaraid_1. Ported buildDeviceLabel with its /dev/disk/by-{id,path}/ prefix stripping and the comma rule, and the scan-time scsi -> auto fallback. Getting this wrong fails silently -- every join in rules/smartctl.yml simply misses. Verified: promtool; a Prometheus fixture covering all three transports plus a megaraid-labelled device, asserting bus and verdict per drive and resolving $ata_device/$sas_device/$nvme_device the way Grafana does; the grid layout; the ExecStart with and without smart.devices; and the collector against this machine's NVMe drive. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ --- rules/smartctl.yml | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) (limited to 'rules/smartctl.yml') diff --git a/rules/smartctl.yml b/rules/smartctl.yml index 3982e06..60a2caa 100644 --- a/rules/smartctl.yml +++ b/rules/smartctl.yml @@ -59,6 +59,36 @@ groups: + sum by (host, device) (smartctl_write_total_uncorrected_errors) or (count by (host, device) (smartctl_device_smart_status) * 0) + # --- NVMe ----------------------------------------------------------------- + # NVMe has neither an ATA attribute table nor SCSI error counters. Its health is + # in the SMART/Health Information log, which smartctl_exporter emits only when the + # interface is nvme. + # + # Each arm is reduced to {host, device} *before* the `or`, so the union + # deduplicates per drive rather than emitting one series per metric name -- the + # count is then "did anything fire", which is all smartctl:health_code asks. + - record: smartctl:nvme_failing:count + expr: | + ( + count by (host, device) (smartctl_device_critical_warning > 0) + or + count by (host, device) (smartctl_device_media_errors > 0) + or + count by (host, device) ( + smartctl_device_available_spare + < on(host, device) smartctl_device_available_spare_threshold + ) + ) + or (count by (host, device) (smartctl_device_smart_status) * 0) + + # percentage_used is the controller's own estimate of consumed write endurance, + # where 100 means the drive has reached its rated lifetime. Drives routinely run + # well past it, so this is a warning and not a failure. + - record: smartctl:nvme_caution:count + expr: | + count by (host, device) (smartctl_device_percentage_used >= 90) + or (count by (host, device) (smartctl_device_smart_status) * 0) + # --- Verdict -------------------------------------------------------------- # 0 good, 1 caution, 2 bad. The inner clamp_max on each group keeps two # simultaneous caution signals from adding up to a false "bad". @@ -69,22 +99,30 @@ groups: (max by (host, device) (smartctl_device_smart_status) == bool 0) + (smartctl:attr_failing:count > bool 0) + (smartctl:scsi_uncorrected:count > bool 0) + + (smartctl:nvme_failing:count > bool 0) , 1) + clamp_max( (smartctl:attr_caution:count > bool 0) + (smartctl:scsi_defects:count > bool 0) + + (smartctl:nvme_caution:count > bool 0) , 1) , 2) # --- Row labels ----------------------------------------------------------- # Which section of the SMART dashboard a drive belongs in. Keyed on metric # presence rather than the `protocol` label: only ATA drives report an attribute - # table, and only SCSI drives report a grown defect list. + # table, only SCSI drives report a grown defect list, and smartctl_exporter emits + # percentage_used only for nvme. The three are disjoint by construction, so the + # order of the `or` arms does not matter -- but a drive matching none of them gets + # no bus, drops out of the smartctl:row join, and silently loses its dashboard + # section, which is what NVMe used to do. - record: smartctl:bus expr: | label_replace(count by (host, device) (smartctl_device_attribute), "bus", "ata", "", "") or label_replace(count by (host, device) (smartctl_scsi_grown_defect_list), "bus", "sas", "", "") + or + label_replace(count by (host, device) (smartctl_device_percentage_used), "bus", "nvme", "", "") # The identity string from collectors/smart-selftest.py, made total so a drive # still gets a row if that collector has not written its file yet. `unless` (not -- cgit v1.3.1