gx10-monitoring

Prometheus + Grafana thermal/GPU telemetry for NVIDIA GB10 nodes (ASUS Ascent GX10 / DGX Spark-class), scaling from a couple of boxes to an 8-node cluster. Every node exposes GPU temp, SoC power, SM clock, utilization, throttle flags, plus board/NVMe/NIC temps — on a single :9100 scrape endpoint, with a ready-made Grafana dashboard.

Built and proven on GB10 nodes running DGX OS; the collectors work regardless of whether you serve models with vLLM, Ollama, or llama.cpp.

What runs on each node

• node_exporter (static arm64 binary — no apt, keeps the DGX OS image clean) on 0.0.0.0:9100, with --collector.disable-defaults and only a light set enabled: textfile thermal_zone hwmon cpu meminfo loadavg filesystem uname time.
  • Why disable-defaults (important on GB10): the default netclass / ethtool collectors crash mid-scrape on the mlx5 (ConnectX-7) driver → the scrape returns "empty reply from server". Disabling defaults fixes that and keeps the footprint light. You still get, for free: the ACPI thermal_zone temps, NVMe temp, and the mlx5 ConnectX-7 sensors via hwmon.
• gx10-gpu-textfile.service — a small nvidia-smi wrapper (gx10-gpu-textfile.sh) re-emitting every 10s into node_exporter's textfile collector. GB10 / DGX OS ships no dcgmi / dcgm-exporter, so nvidia-smi is the supported GPU source. Metrics: gx10_gpu_temp_celsius, gx10_gpu_power_watts (SoC, not wall), gx10_gpu_sm_clock_mhz, gx10_gpu_utilization_ratio, gx10_gpu_throttle_active{reason=...} (hw_thermal / sw_power_cap / sw_thermal / hw_slowdown / hw_power_brake), and gx10_gpu_up.

Install (per node)

Clone the repo on each node and run, as root:

sudo ./install.sh

It downloads node_exporter (arm64), installs the collector + both unit files, creates a node_exporter system user + textfile dir, enables the services, and prints a local scrape check. Override the version with NE_VERSION=1.12.1 sudo -E ./install.sh.

No passwordless sudo on stock DGX OS — run it interactively (or push your key and ssh -t node 'sudo ./install.sh'). For a whole cluster, loop over your node list.

Wire it into Prometheus

Add the nodes as scrape targets — see prometheus-scrape.yaml (plain prometheus.yml, a kube-prometheus-stack ScrapeConfig, or a ServiceMonitor). Set a per-target node label; the dashboard groups by it.

Then import grafana-dashboard.json (or drop it in a ConfigMap labeled grafana_dashboard: "1" if you run the Grafana sidecar).

The dashboard

Top panel is the GPU/SoC hotspot — max(node_thermal_zone_temp{zone=~"0|2"}) per node, with threshold coloring toward the ~96 °C throttle line. Then GPU temp, SoC power, SM clock, utilization, a throttle-reason state timeline, all thermal zones, NVMe/NIC temps, load, and an up/down stat.

GB10 notes worth knowing

• Which thermal zone is the GPU? The 7 acpitz zones are unlabeled. Under a GPU load soak, zones 0 and 2 are the GPU/SoC hotspot — a mirrored pair, hottest, rise most under load, and fall fastest when GPU util hits 0. They read hotter than the nvidia-smi GPU temp (e.g. hotspot ~76 °C while nvidia-smi reported 66 °C on a single-stream inference soak). Design/alert to the hotspot, not the nvidia-smi GPU readout. (Re-verify on your own units — a quick load soak while watching per-zone deltas confirms it.)
• No wall power in software: GB10 has no RAPL/powercap, so gx10_gpu_power_watts is the nvidia-smi SoC figure (~35 W GPU under single-stream inference), not wall draw (~100+ W). Use a smart PDU for true wall power.
• Single-stream inference is thermally light. For real worst-case numbers (chassis/plenum sizing), soak with concurrent/batched load or a compute benchmark, not one stream.
• node_exporter is unauthenticated and bound to 0.0.0.0:9100. Firewall :9100 to your Prometheus source, or bind a specific interface, if the network isn't trusted.

Files

file	purpose
install.sh	per-node installer (downloads node_exporter, installs everything)
gx10-gpu-textfile.sh	nvidia-smi → Prometheus textfile collector
node_exporter.service	node_exporter unit (disable-defaults + light collector set)
gx10-gpu-textfile.service	runs the GPU collector every 10s
prometheus-scrape.yaml	scrape-target examples (plain / operator / ServiceMonitor)
grafana-dashboard.json	importable dashboard (hotspot panel first)