[PR #232] [MERGED] fix(nvml): guard field queries against buffer overruns #230

Closed
opened 2026-10-23 22:05:40 -06:00 by gitea-mirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/XuehaiPan/nvitop/pull/232
Author: @XuehaiPan
Created: 10/22/2026
Status: ✅ Merged
Merged: 10/22/2026
Merged by: @XuehaiPan

Base: main ← Head: fix/nvml-field-values-overrun


📝 Commits (1)

  • e379608 fix(nvml): guard field queries against buffer overruns

📊 Changes

2 files changed (+48 additions, -1 deletions)

View changed files

📝 CHANGELOG.md (+1 -1)
📝 nvitop/api/libnvml.py (+47 -0)

📄 Description

Summary

Fix nvitop-exporter heap corruption during NVLink throughput collection. On driver 595.91.07, nvmlDeviceGetFieldValues() can access an entry beyond the supplied count; a standalone C reproducer confirms the issue independently of the Python bindings.

request N fields → allocate N + 1 entries → query N fields → return N results

Mark the extra guard as unsuccessful so the driver skips it. Initialize all entries defensively, return an empty array without calling NVML for empty input, and warn when the compatibility patch cannot be applied. Update the changelog.

Runtime environment: Ubuntu 22.04.5 LTS, Python 3.10.12, eight NVIDIA H200 GPUs, NVML 13.595.91.07, and nvidia-ml-py 13.610.43.

Evidence

  • Before: free(): invalid next size (normal) abort; Valgrind reports out-of-bounds accesses inside nvmlDeviceGetFieldValues().
  • After: Three exporter collection rounds across all eight GPUs completed with zero Valgrind errors. Sixteen complete HTTP scrapes returned all 144 TX and 144 RX NVLink metrics.
  • Checked the committed code with live NVML queries: 36 successful field results per GPU, plus an empty-input check that forbids native calls.
  • All-file pre-commit checks passed, including Ruff, mypy, and Pylint.

Merge Danger

Door: Two-way; the compatibility wrapper can be reverted without a migration.

Blast Radius: NVML. Applies to field queries through nvitop; requested counts, result lengths, and native errors are preserved for nonempty requests.


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/XuehaiPan/nvitop/pull/232 **Author:** [@XuehaiPan](https://github.com/XuehaiPan) **Created:** 10/22/2026 **Status:** ✅ Merged **Merged:** 10/22/2026 **Merged by:** [@XuehaiPan](https://github.com/XuehaiPan) **Base:** `main` ← **Head:** `fix/nvml-field-values-overrun` --- ### 📝 Commits (1) - [`e379608`](https://github.com/XuehaiPan/nvitop/commit/e379608ffcd4b3958dbc63c2c1b60963662db483) fix(nvml): guard field queries against buffer overruns ### 📊 Changes **2 files changed** (+48 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+1 -1) 📝 `nvitop/api/libnvml.py` (+47 -0) </details> ### 📄 Description ## Summary Fix `nvitop-exporter` heap corruption during NVLink throughput collection. On driver 595.91.07, `nvmlDeviceGetFieldValues()` can access an entry beyond the supplied count; a standalone C reproducer confirms the issue independently of the Python bindings. ```text request N fields → allocate N + 1 entries → query N fields → return N results ``` Mark the extra guard as unsuccessful so the driver skips it. Initialize all entries defensively, return an empty array without calling NVML for empty input, and warn when the compatibility patch cannot be applied. Update the changelog. Runtime environment: Ubuntu 22.04.5 LTS, Python 3.10.12, eight NVIDIA H200 GPUs, NVML 13.595.91.07, and `nvidia-ml-py` 13.610.43. ## Evidence - **Before:** `free(): invalid next size (normal)` abort; Valgrind reports out-of-bounds accesses inside `nvmlDeviceGetFieldValues()`. - **After:** Three exporter collection rounds across all eight GPUs completed with zero Valgrind errors. Sixteen complete HTTP scrapes returned all 144 TX and 144 RX NVLink metrics. - Checked the committed code with live NVML queries: 36 successful field results per GPU, plus an empty-input check that forbids native calls. - All-file pre-commit checks passed, including Ruff, mypy, and Pylint. ## Merge Danger **Door:** Two-way; the compatibility wrapper can be reverted without a migration. **Blast Radius:** NVML. Applies to field queries through nvitop; requested counts, result lengths, and native errors are preserved for nonempty requests. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
github-starred/nvitop#230
No description provided.