add NVMe support, and give the self-test collector the device listHEADmasterflake-modules

Two configurations the modules claimed to support and did not.

NVMe drives had no per-drive section. smartctl:bus routes a drive by metric
presence -- ATA attribute table, or SCSI grown defect list -- and NVMe emits
neither, confirmed against a live smartctl_exporter 0.14.0 on real hardware.
With no bus label the smartctl:row join dropped them, so they appeared in Fleet
Overview and All Drives but could not be opened. Adds an nvme arm keyed on
smartctl_device_percentage_used, verdict rules over the SMART/Health
Information log, and a third repeating section.

The verdict is BAD on a critical warning, media errors, or available spare
below the drive's own threshold, and CAUTION at 90% of rated endurance.
num_err_log_entries is deliberately not in it: the NVMe drive in this machine
is healthy and sits at 2200, because it counts every command that ever
returned an error.

smart.devices reached the exporter but not the self-test collector, which
always ran its own `smartctl --scan` -- exactly what cannot see drives behind a
RAID controller, which is the only reason to set the option. The collector now
takes the same PATH[;TYPE] syntax and passes -d through.

That means it also has to reproduce the exporter's device label, which is not
the basename: /dev/bus/0 as megaraid,1 is bus_0_megaraid_1. Ported
buildDeviceLabel with its /dev/disk/by-{id,path}/ prefix stripping and the
comma rule, and the scan-time scsi -> auto fallback. Getting this wrong fails
silently -- every join in rules/smartctl.yml simply misses.

Verified: promtool; a Prometheus fixture covering all three transports plus a
megaraid-labelled device, asserting bus and verdict per drive and resolving
$ata_device/$sas_device/$nvme_device the way Grafana does; the grid layout;
the ExecStart with and without smart.devices; and the collector against this
machine's NVMe drive.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JKczN1f66kmLxNBLX37voQ

1 files changed, 133 insertions, 21 deletions

diff --git a/dashboards/gen_smart.py b/dashboards/gen_smart.py
index 76ca8fe..d7b0671 100644
--- a/dashboards/gen_smart.py
+++ b/dashboards/gen_smart.py
@@ -9,15 +9,20 @@ it produces:
 A CrystalDiskInfo-style view: a Good/Caution/Bad verdict per drive, the full raw SMART
 attribute table, and self-test history.
 
-The fleet is mixed, so there are two repeating sections:
+The fleet is mixed, so there are three repeating sections, because the three transports
+report health in three unrelated ways:
 
 * SATA drives expose the classic ATA attribute table (value / worst / thresh / raw).
 * SAS drives expose none of it. Their health lives in SCSI error counters instead --
   grown defect list, uncorrected errors, corrected-error breakdown.
+* NVMe drives expose neither, only the SMART/Health Information log -- a critical
+  warning bitfield, media errors, endurance used and the spare block pool.
 
 Drives are routed into the right section by *metric presence* rather than by the
 `protocol` label, which is more robust: only ATA drives have smartctl_device_attribute,
-and only SCSI drives have smartctl_scsi_grown_defect_list.
+only SCSI drives have smartctl_scsi_grown_defect_list, and the exporter emits
+smartctl_device_percentage_used only for nvme. A drive matching none of the three gets
+no section at all, so adding a transport means adding an arm to smartctl:bus.
 
 Metrics used:
 
@@ -63,6 +68,7 @@ PASS_MAP = [{
 HOST = 'host=~"$host"'
 ATA = f'{HOST}, device=~"$ata_device"'
 SAS = f'{HOST}, device=~"$sas_device"'
+NVME = f'{HOST}, device=~"$nvme_device"'
 
 # The per-drive sections come from smartctl:row, one preformatted line per drive.
 #
@@ -149,16 +155,17 @@ panels.append(stat(
     "Healthy", [(f'count(smartctl:health_code{{{HOST}}} == 0) or vector(0)', "drives")],
     gp(0, 1, 4, 4), colormode="background",
     thresholds={"mode": "absolute", "steps": [{"color": "green", "value": None}]},
-    desc="Drives with no failing attribute, no SCSI uncorrected errors and a passing "
-         "SMART self-assessment.",
+    desc="Drives with no failing attribute, no SCSI uncorrected errors, no NVMe "
+         "critical warning and a passing SMART self-assessment.",
 ))
 panels.append(stat(
     "Caution", [(f'count(smartctl:health_code{{{HOST}}} == 1) or vector(0)', "drives")],
     gp(4, 1, 4, 4), colormode="background",
     thresholds={"mode": "absolute", "steps": [
         {"color": "green", "value": None}, {"color": "orange", "value": 1}]},
-    desc="Reallocated / pending / uncorrectable sectors on SATA, or a non-empty grown "
-         "defect list on SAS. Not failing yet — worth watching the trend.",
+    desc="Reallocated / pending / uncorrectable sectors on SATA, a non-empty grown "
+         "defect list on SAS, or 90%+ of rated endurance used on NVMe. Not failing yet "
+         "— worth watching the trend.",
 ))
 panels.append(stat(
     "Failing", [(f'count(smartctl:health_code{{{HOST}}} == 2) or vector(0)', "drives")],
@@ -436,30 +443,133 @@ panels.append(timeseries(
 ))
 
 # ======================================================================================
+# NVMe — repeats once per NVMe drive
+# ======================================================================================
+panels.append(row("${nvme_device:text}", 53, repeat="nvme_device"))
+
+panels.append(stat(
+    "Health", [(f'smartctl:health_code{{{NVME}}}', "{{device}}")],
+    gp(0, 54, 6, 4), mappings=HEALTH_MAP, thresholds=HEALTH_THRESHOLDS,
+    colormode="background",
+))
+panels.append(stat(
+    "Temperature",
+    [(f'smartctl_device_temperature{{{NVME}, temperature_type="current"}}', "temp")],
+    gp(6, 54, 3, 4), unit="celsius",
+    thresholds={"mode": "absolute", "steps": [
+        {"color": "green", "value": None},
+        {"color": "yellow", "value": 50},
+        {"color": "red", "value": 70},
+    ]},
+    desc="NVMe drives run hotter than spinning disks by design and throttle rather "
+         "than fail, so the thresholds here are higher than on the other sections.",
+))
+panels.append(stat(
+    "Powered On", [(power_on(NVME), "hours")],
+    gp(9, 54, 3, 4), unit="s",
+))
+panels.append(stat(
+    # smartctl reports user_capacity only for a single-namespace drive; with several
+    # namespaces the exporter publishes nvme_total_capacity instead and leaves this at
+    # zero, so filter the zero out rather than displaying it.
+    "Capacity",
+    [(f'max by (device) (smartctl_device_capacity_bytes{{{NVME}}} > 0 '
+      f'or smartctl_device_nvme_capacity_bytes{{{NVME}}})', "bytes")],
+    gp(12, 54, 3, 4), unit="bytes",
+))
+panels.append(stat(
+    "Wear", [(f'smartctl_device_percentage_used{{{NVME}}}', "used")],
+    gp(15, 54, 3, 4), unit="percent", colormode="background",
+    thresholds={"mode": "absolute", "steps": [
+        {"color": "green", "value": None},
+        {"color": "yellow", "value": 80},
+        {"color": "red", "value": 100},
+    ]},
+    desc="The controller's estimate of consumed write endurance. 100% means the drive "
+         "has reached its rated lifetime, not that it is about to fail — plenty run "
+         "well past it. Watch how fast it climbs rather than the number.",
+))
+panels.append(stat(
+    "Available Spare", [(f'smartctl_device_available_spare{{{NVME}}}', "remaining")],
+    gp(18, 54, 3, 4), unit="percent", colormode="background",
+    thresholds={"mode": "absolute", "steps": [
+        {"color": "red", "value": None},
+        {"color": "orange", "value": 15},
+        {"color": "green", "value": 25},
+    ]},
+    desc="Spare blocks left for remapping, as a percentage of the factory pool. Falling "
+         "below the drive's own threshold — typically 5–10%, and reported separately as "
+         "smartctl_device_available_spare_threshold — sets a critical warning and is "
+         "treated as a failure here.",
+))
+panels.append(stat(
+    "Media Errors", [(f'smartctl_device_media_errors{{{NVME}}}', "errors")],
+    gp(21, 54, 3, 4), thresholds=OK_BAD, colormode="background",
+    desc="Unrecovered data integrity errors — the NVMe equivalent of SAS uncorrected "
+         "errors. Should be zero.",
+))
+
+panels.append(drive_info(
+    "Drive — $nvme_device", NVME, gp(0, 58, 24, 4),
+    label_cols=["model_name", "serial_number", "firmware_version", "interface",
+                "protocol", "form_factor"],
+    renames={"model_name": "Model", "serial_number": "Serial",
+             "firmware_version": "Firmware", "interface": "Interface",
+             "protocol": "Bus", "form_factor": "Form"},
+    desc="Identity of this drive. NVMe reports no rotation rate, so RPM always reads "
+         "SSD, and no model family or ATA/SATA version.",
+))
+
+panels.append(timeseries(
+    "Wear & Spare — $nvme_device", [
+        (f'smartctl_device_percentage_used{{{NVME}}}', "endurance used"),
+        (f'smartctl_device_available_spare{{{NVME}}}', "spare remaining"),
+        (f'smartctl_device_available_spare_threshold{{{NVME}}}', "spare threshold"),
+    ],
+    gp(0, 62, 12, 8), unit="percent", minval=0,
+    custom={"lineWidth": 2, "fillOpacity": 0},
+    desc="The two lines that matter are spare remaining and its threshold: the drive "
+         "declares itself failing when they cross.",
+))
+panels.append(timeseries(
+    "Errors — $nvme_device", [
+        (f'smartctl_device_critical_warning{{{NVME}}}', "critical warning"),
+        (f'smartctl_device_media_errors{{{NVME}}}', "media errors"),
+        (f'smartctl_device_num_err_log_entries{{{NVME}}}', "error log entries"),
+    ],
+    gp(12, 62, 12, 8), minval=0,
+    custom={"lineWidth": 2, "fillOpacity": 0, "showPoints": "auto"},
+    desc="Critical warning is a bitfield — any non-zero value is the drive asking to be "
+         "replaced. Error log entries count every command that ever returned an error, "
+         "including entirely harmless ones: a healthy consumer SSD can sit at a few "
+         "thousand. It is context, not a fault, which is why the verdict ignores it.",
+))
+
+# ======================================================================================
 # Self-tests
 # ======================================================================================
-panels.append(row("Self-Tests", 53))
+panels.append(row("Self-Tests", 70))
 
 panels.append(stat(
     "Drives Never Tested",
     [(f'count(smartctl_device_smart_status{{{HOST}}}) - '
       f'count(count by (device) (smart_selftest_last_lifetime_hours{{{HOST}}})) or '
       f'count(smartctl_device_smart_status{{{HOST}}})', "drives")],
-    gp(0, 54, 4, 5), thresholds=OK_BAD, colormode="background",
+    gp(0, 71, 4, 5), thresholds=OK_BAD, colormode="background",
     desc="Drives with no self-test log entries at all. smartd only runs tests if its "
          "config includes an -s schedule; merely enabling the service does not.",
 ))
 panels.append(stat(
     "Failed Self-Tests",
     [(f'count(smart_selftest_last_passed{{{HOST}}} == 0) or vector(0)', "tests")],
-    gp(4, 54, 4, 5), thresholds=OK_BAD, colormode="background",
+    gp(4, 71, 4, 5), thresholds=OK_BAD, colormode="background",
 ))
 panels.append(stat(
     "Oldest Long Test",
     [(f'max({power_on()} / 3600 - on(device) group_right() '
       f'smart_selftest_last_lifetime_hours{{{HOST}, test_type="extended"}})',
       "hours ago")],
-    gp(8, 54, 4, 5), unit="h",
+    gp(8, 71, 4, 5), unit="h",
     thresholds={"mode": "absolute", "steps": [
         {"color": "green", "value": None},
         {"color": "yellow", "value": 24 * 14},
@@ -495,7 +605,7 @@ panels.append(table(
         (_keyed(power_on() + ' / 3600 - on(device) group_right() '
                 f'smart_selftest_last_lifetime_hours{{{HOST}}}'), None),
     ],
-    gp(12, 54, 12, 10), keep="drive_test|device|test_type|Value.*",
+    gp(12, 71, 12, 10), keep="drive_test|device|test_type|Value.*",
     renames={
         "__join__": "drive_test", "device": "Device", "test_type": "Test",
         "Value #A": "Last Result", "Value #B": "At Hours",
@@ -523,26 +633,26 @@ panels.append(table(
 # ======================================================================================
 # Collapsed: trends and diagnostics
 # ======================================================================================
-panels.append(row("Attribute Trends", 64, collapsed=True, children=[
+panels.append(row("Attribute Trends", 81, collapsed=True, children=[
     timeseries(
         "Reallocated Sectors (all SATA drives)",
         [(f'smartctl_device_attribute{{{HOST}, attribute_id="5", '
           f'attribute_value_type="raw"}}', "{{device}}")],
-        gp(0, 65, 8, 7), minval=0,
+        gp(0, 82, 8, 7), minval=0,
         custom={"lineWidth": 2, "fillOpacity": 0, "showPoints": "auto"},
     ),
     timeseries(
         "Pending & Uncorrectable Sectors",
         [(f'smartctl_device_attribute{{{HOST}, attribute_id=~"197|198", '
           f'attribute_value_type="raw"}}', "{{device}} {{attribute_name}}")],
-        gp(8, 65, 8, 7), minval=0,
+        gp(8, 82, 8, 7), minval=0,
         custom={"lineWidth": 2, "fillOpacity": 0, "showPoints": "auto"},
     ),
     timeseries(
         "CRC Errors (cabling)",
         [(f'smartctl_device_attribute{{{HOST}, attribute_id="199", '
           f'attribute_value_type="raw"}}', "{{device}}")],
-        gp(16, 65, 8, 7), minval=0,
+        gp(16, 82, 8, 7), minval=0,
         custom={"lineWidth": 2, "fillOpacity": 0, "showPoints": "auto"},
         desc="Attribute 199 counts errors on the SATA link rather than the platters. "
              "Growth here usually means a cable or backplane, not a dying disk.",
@@ -552,21 +662,21 @@ panels.append(row("Attribute Trends", 64, collapsed=True, children=[
             (f'smartctl_device_bytes_written{{{HOST}}}', "{{device}} written"),
             (f'smartctl_device_bytes_read{{{HOST}}}', "{{device}} read"),
         ],
-        gp(0, 72, 12, 7), unit="bytes", minval=0,
+        gp(0, 89, 12, 7), unit="bytes", minval=0,
     ),
     timeseries(
         "Health Code History",
         [(f'smartctl:health_code{{{HOST}}}', "{{device}}")],
-        gp(12, 72, 12, 7), mappings=HEALTH_MAP, minval=0, maxval=2,
+        gp(12, 89, 12, 7), mappings=HEALTH_MAP, minval=0, maxval=2,
         custom={"lineWidth": 2, "fillOpacity": 0, "lineInterpolation": "stepAfter"},
     ),
 ]))
 
-panels.append(row("Collector Health", 65, collapsed=True, children=[
+panels.append(row("Collector Health", 82, collapsed=True, children=[
     timeseries(
         "smartctl Exit Status",
         [(f'smartctl_device_smartctl_exit_status{{{HOST}}}', "{{device}}")],
-        gp(0, 66, 8, 7), minval=0,
+        gp(0, 83, 8, 7), minval=0,
         desc="Non-zero means smartctl itself had trouble reading the device. Bit 0 is "
              "a command-line problem, bit 1 a device-open failure, bit 3 a failing "
              "drive.",
@@ -574,14 +684,14 @@ panels.append(row("Collector Health", 65, collapsed=True, children=[
     timeseries(
         "Self-Test Collector Duration",
         [(f'smart_selftest_collection_duration_seconds{{{HOST}}}', "collection")],
-        gp(8, 66, 8, 7), unit="s", minval=0,
+        gp(8, 83, 8, 7), unit="s", minval=0,
     ),
     timeseries(
         "Devices Seen", [
             (f'smartctl_devices{{{HOST}}}', "exporter"),
             (f'smart_selftest_devices{{{HOST}}}', "self-test collector"),
         ],
-        gp(16, 66, 8, 7), minval=0,
+        gp(16, 83, 8, 7), minval=0,
         desc="These should agree. A divergence means one of the two cannot see some "
              "drives.",
     ),
@@ -610,6 +720,8 @@ print(json.dumps(dashboard(
                   label="SATA drive", regex=ROW_REGEX),
         query_var("sas_device", DEVICE_QUERY.format(bus="sas"),
                   label="SAS drive", regex=ROW_REGEX),
+        query_var("nvme_device", DEVICE_QUERY.format(bus="nvme"),
+                  label="NVMe drive", regex=ROW_REGEX),
     ],
     tags=["smart", "storage", "hardware"],
 ), indent=2))