[GH-ISSUE #236] [BUG] Process %CPU oscillates between 0% and the real value when the process-metrics screen or treeview is open #235

New issue

Closed

opened 2026-10-31 22:03:33 -06:00 by gitea-mirror · 0 comments

gitea-mirror commented

2026-10-31 22:03:33 -06:00

Owner

Originally created by @arinawang11 on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/236

Originally assigned to: @XuehaiPan on GitHub.

What version of nvitop are you using?

1.7.1. The same code structure is present in 1.7.2 and on main (verified by reading the sources).

Operating system and version

Windows Server 2025 Datacenter 10.0.26100 (96 logical CPUs).

NVIDIA driver version

581.80

NVIDIA-SMI

Sat Oct 31 15:34:24 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 581.80                 Driver Version: 581.80         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Graphics Device       TCC   |   00000000:10:00.0 Off |                    0 |
| N/A   45C    P0            492W /  500W |    1836MiB /  49152MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA T4G                   WDDM  |   00000000:2C:00.0 Off |                    0 |
| N/A   69C    P0             70W /   70W |    5524MiB /  15360MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA DRIVE-PG199-PROD      TCC   |   00000000:60:00.0 Off |                    0 |
| N/A   62C    P0            320W /  N/A  |    1734MiB /  32768MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           26144      C   ...ulti-3-7-6\SRBMiner-MULTI.exe       1826MiB |
|    1   N/A  N/A            4136    C+G   C:\Windows\System32\LogonUI.exe       N/A      |
|    1   N/A  N/A            4152    C+G   C:\Windows\System32\dwm.exe           N/A      |
|    1   N/A  N/A            5240    C+G   ..._cw5n1h2txyewy\SearchHost.exe      N/A      |
|    1   N/A  N/A            8356    C+G   C:\Windows\System32\dwm.exe           N/A      |
|    1   N/A  N/A           10296    C+G   ...indows\System32\ShellHost.exe      N/A      |
|    1   N/A  N/A           13616    C+G   C:\Windows\explorer.exe               N/A      |
|    1   N/A  N/A           15072    C+G   ...t\Edge\Application\msedge.exe      N/A      |
|    1   N/A  N/A           18036    C+G   ...y\StartMenuExperienceHost.exe      N/A      |
|    1   N/A  N/A           21080    C+G   ...es\ikuuu_vpn\app\iKuuuVPN.exe      N/A      |
|    1   N/A  N/A           21468    C+G   ...rpLauncher\SakuraLauncher.exe      N/A      |
|    1   N/A  N/A           21556    C+G   ...yb3d8bbwe\WindowsTerminal.exe      N/A      |
|    1   N/A  N/A           22028    C+G   ...9041.0\win-x64\MsiPsu.App.exe      N/A      |
|    1   N/A  N/A           22744    C+G   ...xyewy\ShellExperienceHost.exe      N/A      |
|    1   N/A  N/A           25072    C+G   ...t\Edge\Application\msedge.exe      N/A      |
|    1   N/A  N/A           26144      C   ...ulti-3-7-6\SRBMiner-MULTI.exe      N/A      |
|    1   N/A  N/A           29916    C+G   ...5n1h2txyewy\TextInputHost.exe      N/A      |
|    1   N/A  N/A           33000    C+G   ...041.0\win-x64\NaiV5Studio.exe      N/A      |
|    1   N/A  N/A           34604    C+G   C:\Windows\explorer.exe               N/A      |
|    1   N/A  N/A           37320    C+G   C:\Windows\System32\WUDFHost.exe      N/A      |
|    2   N/A  N/A           26144      C   ...ulti-3-7-6\SRBMiner-MULTI.exe       1724MiB |
+-----------------------------------------------------------------------------------------+

Python environment

Installed with pip.

3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] win32
colorama==0.4.6
nvidia-ml-py==13.590.48
nvitop==1.7.1
psutil==7.2.2
windows-curses==2.4.2

Problem description

Expected: the %CPU value of a process is stable. A process pinning one CPU core should report ~100% on every refresh.

Actual: while the process-metrics screen is open (Enter on a process) or the treeview is open (t), the CPU percentage of the process alternates between 0.0% and its real value on every sample. The CPU history graph on the metrics screen looks like a square wave. Going back to the main screen (Enter / Esc) makes the value stable again.

Root cause: nvitop caches HostProcess per PID and GpuProcess per (pid, device), while multiple snapshot daemons call psutil's non-blocking Process.cpu_percent() on the same object concurrently. That API is stateful: it returns a delta since the previous call on that object. The loser of the race measures a sub-millisecond window; on Windows GetProcessTimes is only updated every ~15.6 ms, so it reads a zero delta and returns exactly 0.0%. The winner measures the full window and returns the real value. The same pattern is reproducible on any platform where the CPU-time counter has a coarser granularity than the concurrent sampling window.

Relevant code:

nvitop/api/process.py:196-230 — HostProcess.INSTANCES cache by PID; the same instance is shared by all screens.
nvitop/api/process.py:834-847 — GpuProcess.cpu_percent() delegates to self.host.cpu_percent().
nvitop/tui/screens/metrics.py:294-316 — the process-metrics daemon samples the same cached GpuProcess every 0.5 s.
nvitop/tui/screens/main/panels/process.py:362-388 — the main process panel daemon; its thread is only stopped in destroy() (lines 662-664).
nvitop/tui/library/displayable.py:143-149 and nvitop/tui/tui.py:316-322 — hiding a screen only sets visible = False; the panel daemon keeps running, so both daemons sample the same object while the metrics/treeview screen is shown.
nvitop/tui/screens/treeview.py:103 — treeview re-samples shared HostProcess/GpuProcess instances instead of reusing Snapshot.cpu_percent.

Steps to Reproduce

Minimal reproduction, no GPU required:

import subprocess, sys, threading, time
from nvitop.api import HostProcess

child = subprocess.Popen([sys.executable, '-c', 'while True: pass'])
time.sleep(0.5)
p = HostProcess(child.pid)   # cached singleton, shared by all nvitop screens
results = []

def worker(tag):
    for _ in range(6):
        results.append((tag, round(float(p.cpu_percent()), 1)))
        time.sleep(0.5)

threads = [threading.Thread(target=worker, args=(t,)) for t in ('thread-1', 'thread-2')]
[t.start() for t in threads]
[t.join() for t in threads]
child.kill()
print(results)

Output on Windows Server 2025 with psutil 7.2.2:

thread-1=  0.0%   thread-2=  0.0%
thread-1= 97.1%   thread-2=  0.0%
thread-1=100.0%   thread-2=  0.0%
thread-1=100.0%   thread-2=  0.0%
thread-2=100.0%   thread-1=  0.0%
thread-1=100.0%   thread-2=  0.0%

With a single sampler, the same process reports 100.0% on every sample.

TUI steps:

Run nvitop on a machine with a GPU process that is CPU-active.
Select the process and press Enter to open the process-metrics screen. The CPU history graph alternates between the real value and 0.
Press Enter / Esc to return to the main screen. The value becomes stable again.
Press t to open treeview. The %CPU column of the GPU processes alternates as well.

Traceback

None.

Additional context

Also reproducible through the public API by calling GpuProcess.as_snapshot() / HostProcess.cpu_percent() from two threads.
Long-standing issue: the singleton caches and the process-metrics daemon already exist in v0.9.0; the same structure is present in v1.7.2 and main.
Not hardware-specific: any NVIDIA GPU/driver combination can be affected; only CPU-active processes show the oscillation (idle processes read 0% from both callers).
Suggested minimal fix: pause the main screen panels' snapshot daemons while the main screen is not visible, and make treeview reuse Snapshot.cpu_percent for processes that already have a snapshot. A more general fix would serialize per-PID sampling with a short cache in HostProcess.cpu_percent.

Originally created by @arinawang11 on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/XuehaiPan/nvitop/issues/236 Originally assigned to: @XuehaiPan on GitHub. ## What version of nvitop are you using? 1.7.1. The same code structure is present in 1.7.2 and on `main` (verified by reading the sources). ## Operating system and version Windows Server 2025 Datacenter 10.0.26100 (96 logical CPUs). ## NVIDIA driver version 581.80 ## NVIDIA-SMI ```text Sat Oct 31 15:34:24 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 581.80 Driver Version: 581.80 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA Graphics Device TCC | 00000000:10:00.0 Off | 0 | | N/A 45C P0 492W / 500W | 1836MiB / 49152MiB | 100% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA T4G WDDM | 00000000:2C:00.0 Off | 0 | | N/A 69C P0 70W / 70W | 5524MiB / 15360MiB | 100% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA DRIVE-PG199-PROD TCC | 00000000:60:00.0 Off | 0 | | N/A 62C P0 320W / N/A | 1734MiB / 32768MiB | 100% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 26144 C ...ulti-3-7-6\SRBMiner-MULTI.exe 1826MiB | | 1 N/A N/A 4136 C+G C:\Windows\System32\LogonUI.exe N/A | | 1 N/A N/A 4152 C+G C:\Windows\System32\dwm.exe N/A | | 1 N/A N/A 5240 C+G ..._cw5n1h2txyewy\SearchHost.exe N/A | | 1 N/A N/A 8356 C+G C:\Windows\System32\dwm.exe N/A | | 1 N/A N/A 10296 C+G ...indows\System32\ShellHost.exe N/A | | 1 N/A N/A 13616 C+G C:\Windows\explorer.exe N/A | | 1 N/A N/A 15072 C+G ...t\Edge\Application\msedge.exe N/A | | 1 N/A N/A 18036 C+G ...y\StartMenuExperienceHost.exe N/A | | 1 N/A N/A 21080 C+G ...es\ikuuu_vpn\app\iKuuuVPN.exe N/A | | 1 N/A N/A 21468 C+G ...rpLauncher\SakuraLauncher.exe N/A | | 1 N/A N/A 21556 C+G ...yb3d8bbwe\WindowsTerminal.exe N/A | | 1 N/A N/A 22028 C+G ...9041.0\win-x64\MsiPsu.App.exe N/A | | 1 N/A N/A 22744 C+G ...xyewy\ShellExperienceHost.exe N/A | | 1 N/A N/A 25072 C+G ...t\Edge\Application\msedge.exe N/A | | 1 N/A N/A 26144 C ...ulti-3-7-6\SRBMiner-MULTI.exe N/A | | 1 N/A N/A 29916 C+G ...5n1h2txyewy\TextInputHost.exe N/A | | 1 N/A N/A 33000 C+G ...041.0\win-x64\NaiV5Studio.exe N/A | | 1 N/A N/A 34604 C+G C:\Windows\explorer.exe N/A | | 1 N/A N/A 37320 C+G C:\Windows\System32\WUDFHost.exe N/A | | 2 N/A N/A 26144 C ...ulti-3-7-6\SRBMiner-MULTI.exe 1724MiB | +-----------------------------------------------------------------------------------------+ ``` ## Python environment Installed with `pip`. ```text 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] win32 colorama==0.4.6 nvidia-ml-py==13.590.48 nvitop==1.7.1 psutil==7.2.2 windows-curses==2.4.2 ``` ## Problem description **Expected:** the `%CPU` value of a process is stable. A process pinning one CPU core should report ~100% on every refresh. **Actual:** while the process-metrics screen is open (`Enter` on a process) or the treeview is open (`t`), the CPU percentage of the process alternates between `0.0%` and its real value on every sample. The CPU history graph on the metrics screen looks like a square wave. Going back to the main screen (`Enter` / `Esc`) makes the value stable again. **Root cause:** `nvitop` caches `HostProcess` per PID and `GpuProcess` per `(pid, device)`, while multiple snapshot daemons call psutil's *non-blocking* `Process.cpu_percent()` on the same object concurrently. That API is stateful: it returns a delta since the previous call on that object. The loser of the race measures a sub-millisecond window; on Windows `GetProcessTimes` is only updated every ~15.6 ms, so it reads a zero delta and returns exactly `0.0%`. The winner measures the full window and returns the real value. The same pattern is reproducible on any platform where the CPU-time counter has a coarser granularity than the concurrent sampling window. Relevant code: - `nvitop/api/process.py:196-230` — `HostProcess.INSTANCES` cache by PID; the same instance is shared by all screens. - `nvitop/api/process.py:834-847` — `GpuProcess.cpu_percent()` delegates to `self.host.cpu_percent()`. - `nvitop/tui/screens/metrics.py:294-316` — the process-metrics daemon samples the same cached `GpuProcess` every 0.5 s. - `nvitop/tui/screens/main/panels/process.py:362-388` — the main process panel daemon; its thread is only stopped in `destroy()` (lines 662-664). - `nvitop/tui/library/displayable.py:143-149` and `nvitop/tui/tui.py:316-322` — hiding a screen only sets `visible = False`; the panel daemon keeps running, so both daemons sample the same object while the metrics/treeview screen is shown. - `nvitop/tui/screens/treeview.py:103` — treeview re-samples shared `HostProcess`/`GpuProcess` instances instead of reusing `Snapshot.cpu_percent`. ## Steps to Reproduce Minimal reproduction, no GPU required: ```python import subprocess, sys, threading, time from nvitop.api import HostProcess child = subprocess.Popen([sys.executable, '-c', 'while True: pass']) time.sleep(0.5) p = HostProcess(child.pid) # cached singleton, shared by all nvitop screens results = [] def worker(tag): for _ in range(6): results.append((tag, round(float(p.cpu_percent()), 1))) time.sleep(0.5) threads = [threading.Thread(target=worker, args=(t,)) for t in ('thread-1', 'thread-2')] [t.start() for t in threads] [t.join() for t in threads] child.kill() print(results) ``` Output on Windows Server 2025 with psutil 7.2.2: ```text thread-1= 0.0% thread-2= 0.0% thread-1= 97.1% thread-2= 0.0% thread-1=100.0% thread-2= 0.0% thread-1=100.0% thread-2= 0.0% thread-2=100.0% thread-1= 0.0% thread-1=100.0% thread-2= 0.0% ``` With a single sampler, the same process reports `100.0%` on every sample. TUI steps: 1. Run `nvitop` on a machine with a GPU process that is CPU-active. 2. Select the process and press `Enter` to open the process-metrics screen. The CPU history graph alternates between the real value and 0. 3. Press `Enter` / `Esc` to return to the main screen. The value becomes stable again. 4. Press `t` to open treeview. The `%CPU` column of the GPU processes alternates as well. ## Traceback None. ## Additional context - Also reproducible through the public API by calling `GpuProcess.as_snapshot()` / `HostProcess.cpu_percent()` from two threads. - Long-standing issue: the singleton caches and the process-metrics daemon already exist in v0.9.0; the same structure is present in v1.7.2 and `main`. - Not hardware-specific: any NVIDIA GPU/driver combination can be affected; only CPU-active processes show the oscillation (idle processes read 0% from both callers). - Suggested minimal fix: pause the main screen panels' snapshot daemons while the main screen is not visible, and make treeview reuse `Snapshot.cpu_percent` for processes that already have a snapshot. A more general fix would serialize per-PID sampling with a short cache in `HostProcess.cpu_percent`.

gitea-mirror

2026-10-31 22:03:33 -06:00