github-starred/RhapsodiOS

Watch

Fork 0

mirror of https://github.com/RhapsodiOS/RhapsodiOS.git synced 2026-11-03 22:01:40 -07:00

[GH-ISSUE #78] nfs: unmount of the loopback NFS scratch mount is busy after a full stress run, and umount -f hangs #72

New issue

Closed

opened 2026-11-03 22:03:44 -07:00 by gitea-mirror · 3 comments

gitea-mirror commented

2026-11-03 22:03:44 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/78

In the combined retest of ffs-perf f68d4802c, the first all-category ffsbench stress --fs nfs run ended with umount returning "Device busy". umount -f then hung until the harness killed it (vm/stress_fs.sh lim 120).

There is no earlier all-category NFS run to compare with. Earlier fs+mmap NFS runs unmounted cleanly.

Find what still holds the mount (leftover processes or cwd from the stress run, open or cached vnodes, a v_usecount leak in nfs from one of the categories), and why the forced unmount hangs instead of failing or succeeding.

Logs: vm/work/ffsbench/runs/combined-nfsmsdos, vm/work/combined-retest/.

Originally created by @raynorpat on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/78 In the combined retest of ffs-perf f68d4802c, the first all-category `ffsbench stress --fs nfs` run ended with `umount` returning "Device busy". `umount -f` then hung until the harness killed it (`vm/stress_fs.sh` `lim 120`). There is no earlier all-category NFS run to compare with. Earlier fs+mmap NFS runs unmounted cleanly. Find what still holds the mount (leftover processes or cwd from the stress run, open or cached vnodes, a v_usecount leak in nfs from one of the categories), and why the forced unmount hangs instead of failing or succeeding. Logs: `vm/work/ffsbench/runs/combined-nfsmsdos`, `vm/work/combined-retest/`.

gitea-mirror

2026-11-03 22:03:44 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:03:45 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 21, 2026):

Fixed by e9e67317f and 249ceb8fb, merged into ffs-perf in 5342b6d9e. Both causes were kernel bugs.

What held the mount: a loopback NFS self-deadlock.

An nfsd serving a read or write holds the exported UFS file's vnode lock inside mapfs_io. When the mapfs map or file cache is full, remap_vnode (kern/mapfs.c ~577) or mapfs_cache_trim (~937) frees the head of the shared LRU.
When that entry is an NFS client file, freeing it waits on the NFS server: vnode_pageout → nfs_asyncio ("nfsaio", nfs_bio.c :1031), or vrele → nfs_inactive → nfs_vinvalbuf ("vinvalbuf", nfs_node.c :265). The server is the nfsd itself, often holding the very file's lock.
The other nfsds then queue on that inode lock. The stuck nfs_inactive of a silly-renamed .nfs file holds a root vnode reference, hence "Device busy".
On the combined kernel the all-category run deadlocked in 2 of 4 runs.

Fix e9e67317f: a caller working on a local file never picks an NFS entry to free. For an NFS entry that still holds map space it frees only the map window (mapfs_window_free: no push, the object and its dirty pages stay cached). vmp_push falls back to the tracked dirty range when the window is gone.

umount -f: nfs_unmount returned EBUSY on a busy root even with MNT_FORCE (nfs_vfsops.c :936). That was the second "Device busy". Fix 249ceb8fb skips that check under MNT_FORCE.

Verified on the fixed kernel (cksum 1353083474):

All-category NFS harness runs: 3 (seeds 75, 75, 77), plus 2 single-mount runs on the guest, 0 deadlocks, every unmount clean.
A forced unmount with a busy file and a cwd on the mount root succeeds, and remounting works.
make check-fs and check-net pass on UFS and with -d /mnt/nfs.

Left open:

Forcing an unmount off a server that is really hung still blocks. umount(8) stats the mount point, dounmount calls VFS_SYNC(MNT_WAIT) even under MNT_FORCE, and NFS requests can't be cancelled.
When only NFS entries are left, a local caller's trim stops early, so the mapped-file count can go over mfs_files_max until an NFS caller trims.

@raynorpat commented on GitHub (Oct 21, 2026): Fixed by e9e67317f and 249ceb8fb, merged into ffs-perf in 5342b6d9e. Both causes were kernel bugs. **What held the mount: a loopback NFS self-deadlock.** - An nfsd serving a read or write holds the exported UFS file's vnode lock inside `mapfs_io`. When the mapfs map or file cache is full, `remap_vnode` (`kern/mapfs.c` ~577) or `mapfs_cache_trim` (~937) frees the head of the shared LRU. - When that entry is an NFS client file, freeing it waits on the NFS server: `vnode_pageout` → `nfs_asyncio` ("nfsaio", `nfs_bio.c` :1031), or `vrele` → `nfs_inactive` → `nfs_vinvalbuf` ("vinvalbuf", `nfs_node.c` :265). The server is the nfsd itself, often holding the very file's lock. - The other nfsds then queue on that inode lock. The stuck `nfs_inactive` of a silly-renamed `.nfs` file holds a root vnode reference, hence "Device busy". - On the combined kernel the all-category run deadlocked in 2 of 4 runs. **Fix e9e67317f:** a caller working on a local file never picks an NFS entry to free. For an NFS entry that still holds map space it frees only the map window (`mapfs_window_free`: no push, the object and its dirty pages stay cached). `vmp_push` falls back to the tracked dirty range when the window is gone. **umount -f:** `nfs_unmount` returned EBUSY on a busy root even with MNT_FORCE (`nfs_vfsops.c` :936). That was the second "Device busy". Fix 249ceb8fb skips that check under MNT_FORCE. **Verified** on the fixed kernel (cksum 1353083474): - All-category NFS harness runs: 3 (seeds 75, 75, 77), plus 2 single-mount runs on the guest, 0 deadlocks, every unmount clean. - A forced unmount with a busy file and a cwd on the mount root succeeds, and remounting works. - `make check-fs` and `check-net` pass on UFS and with `-d /mnt/nfs`. **Left open:** - Forcing an unmount off a server that is really hung still blocks. umount(8) stats the mount point, `dounmount` calls `VFS_SYNC(MNT_WAIT)` even under MNT_FORCE, and NFS requests can't be cancelled. - When only NFS entries are left, a local caller's trim stops early, so the mapped-file count can go over `mfs_files_max` until an NFS caller trims.

gitea-mirror commented

2026-11-03 22:03:46 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 22, 2026):

Reopened after the final retest of ffs-perf 966dc5243 (kernel cksum 3569582201). Two problems remain.

1. A kernel panic in mapfs_sync, probably from e9e67317f.

Seen once in the full five-filesystem run. The NFS target ended with a plain umount "Device busy", then a forced unmount succeeded. Shortly after, while msdos vm was running, update (pid 45) faulted in sync: trap 0xe, err=2, eip 0x1d0c98 (_mapfs_sync+0x94). That is the inlined vmp_get dequeuing from vm_info_queue, with a next pointer of 0x41140000.
The queue's first entry (0x818afa0) is freed memory reused by another zone. The newest entry has va=0 and size=0, the state mapfs_window_free leaves.
Suspected: mapfs_window_free drops its locks and later re-enqueues the entry. It checks mapped, queued and use_count, but not that the entry is being freed, so vm_info_free can run in the gap and a freed entry goes back on the queue. mapfs_sync also reads an entry after unlocking without holding a reference.
6 targeted tries didn't reproduce it.
Evidence: vm/work/final-retest/panic/ (bt.txt, qback.txt, serial-final.log).

2. The plain NFS unmount is still "Device busy" after a full all-category run. The forced unmount now succeeds. Something still holds a reference on the mount after the deadlock fix.

Runs: vm/work/ffsbench/runs/final*.

@raynorpat commented on GitHub (Oct 22, 2026): Reopened after the final retest of ffs-perf 966dc5243 (kernel cksum 3569582201). Two problems remain. **1. A kernel panic in `mapfs_sync`, probably from e9e67317f.** - Seen once in the full five-filesystem run. The NFS target ended with a plain `umount` "Device busy", then a forced unmount succeeded. Shortly after, while msdos `vm` was running, `update` (pid 45) faulted in `sync`: trap 0xe, err=2, eip 0x1d0c98 (`_mapfs_sync+0x94`). That is the inlined `vmp_get` dequeuing from `vm_info_queue`, with a next pointer of 0x41140000. - The queue's first entry (0x818afa0) is freed memory reused by another zone. The newest entry has va=0 and size=0, the state `mapfs_window_free` leaves. - Suspected: `mapfs_window_free` drops its locks and later re-enqueues the entry. It checks `mapped`, `queued` and `use_count`, but not that the entry is being freed, so `vm_info_free` can run in the gap and a freed entry goes back on the queue. `mapfs_sync` also reads an entry after unlocking without holding a reference. - 6 targeted tries didn't reproduce it. - Evidence: `vm/work/final-retest/panic/` (`bt.txt`, `qback.txt`, `serial-final.log`). **2. The plain NFS unmount is still "Device busy"** after a full all-category run. The forced unmount now succeeds. Something still holds a reference on the mount after the deadlock fix. Runs: `vm/work/ffsbench/runs/final*`.

gitea-mirror commented

2026-11-03 22:03:47 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 30, 2026):

The two problems from the reopen are fixed by 387d97d67 and dc5864824, merged into ffs-perf in 3d017f064. They share one root cause.

Why the plain unmount was busy (line numbers from 966dc5243):

unmap_vnode holds a temporary map_count++ across a blocking VOP_GETATTR and push (mapfs.c :491-495, :510/:524). mapfs_trunc does the same (:743/:749).
An open made before the file had a pager never took a count (vfs_vnops.c :285), but its close still decrements.
When every t_shared worker exits at once over NFS, one close takes another's temporary hold, and the count drops to -1 or -2.
mapfs_cache_clear only frees entries with map_count == 0 (:1028). These silly-renamed .nfs files therefore kept their vnode, and with it a root reference: root use 3, so nfs_unmount returned EBUSY. gdbstub on the stock kernel showed map -1 and map -2 on two .nfsA… vnodes.
Fix dc5864824: the three temporary decrements never go below zero.

Why mapfs_sync panicked:

vm_info_free (:315-340) calls mapfs_uncache, which skips memfree when map_count != 0, then zfrees an entry that is still on vm_info_queue.
A forced unmount reclaims exactly those files: open, or with a negative count left by the bug above.
This predates e9e67317f; 249ceb8fb only let umount -f get past the busy root and reach it.
The suspected mapfs_window_free re-enqueue was also a real hole, but it isn't what fired.
Fix 387d97d67: vm_info_free dequeues the entry under the lock. If a holder still has it (use_count > 0), a new orphan bit leaves the free to that holder's last vmp_put. Neither vmp_put nor mapfs_window_free re-enqueues a dying entry.

Verified on the kernel built from dc5864824 (cksum 1631582623):

Panic probe: a kernel that panics on a freed or dying queued entry, running exec from NFS, holding it open, umount -f, then sync. It panicked in round 1 without the fix (vflush → vm_info_free) and passed 9/9 with it. A write+sync variant also passed 9/9.
Same recipe on the stock kernel (no probe): it leaves a freed, dying, queued entry. On the final kernel, 3 runs: no panic, queue clean.
Busy mount: check-mmap -d /mnt/nfs then a plain umount went from Device busy 3/3 to success 3/3.
Checks: check-fs, check-mmap and check-vm pass on UFS and on /mnt/nfs.
Full runs: two all-category five-filesystem runs (seed 75): no panic, NFS plain unmount succeeds, every fsck clean. Their remaining FAILs are the known fs rw and proc pty failures, also seen on the stock kernel, plus a run-2 UFS phase that ran about 3x slow (likely host load).

Left open:

The negative count: never caught at the exact decrement; a negative-count probe kernel wasn't built.
Paths without the vnode lock: those that reach a vm_info without it (unmap_vnode across VOP_GETATTR, vmp_invalidate, and mapfs_sync between its unlock and vmp_get) can still touch a freed entry if a forced unmount races them.
vmp_put that drops use_count to 0: vm_info_free can free the entry in the short gap before that vmp_put releases vmp->lock.
Forced reclaim of an open file: it still leaks its mfs window and object reference, as the stock kernel does.

@raynorpat commented on GitHub (Oct 30, 2026): The two problems from the reopen are fixed by 387d97d67 and dc5864824, merged into ffs-perf in 3d017f064. They share one root cause. **Why the plain unmount was busy** (line numbers from 966dc5243): - `unmap_vnode` holds a temporary `map_count++` across a blocking `VOP_GETATTR` and push (`mapfs.c` :491-495, :510/:524). `mapfs_trunc` does the same (:743/:749). - An open made before the file had a pager never took a count (`vfs_vnops.c` :285), but its close still decrements. - When every `t_shared` worker exits at once over NFS, one close takes another's temporary hold, and the count drops to -1 or -2. - `mapfs_cache_clear` only frees entries with `map_count == 0` (:1028). These silly-renamed `.nfs` files therefore kept their vnode, and with it a root reference: root use 3, so `nfs_unmount` returned EBUSY. gdbstub on the stock kernel showed `map -1` and `map -2` on two `.nfsA…` vnodes. - **Fix dc5864824:** the three temporary decrements never go below zero. **Why `mapfs_sync` panicked:** - `vm_info_free` (:315-340) calls `mapfs_uncache`, which skips `memfree` when `map_count != 0`, then `zfree`s an entry that is still on `vm_info_queue`. - A forced unmount reclaims exactly those files: open, or with a negative count left by the bug above. - This predates e9e67317f; 249ceb8fb only let `umount -f` get past the busy root and reach it. - The suspected `mapfs_window_free` re-enqueue was also a real hole, but it isn't what fired. - **Fix 387d97d67:** `vm_info_free` dequeues the entry under the lock. If a holder still has it (`use_count > 0`), a new `orphan` bit leaves the free to that holder's last `vmp_put`. Neither `vmp_put` nor `mapfs_window_free` re-enqueues a dying entry. **Verified** on the kernel built from dc5864824 (cksum 1631582623): - **Panic probe:** a kernel that panics on a freed or dying queued entry, running exec from NFS, holding it open, `umount -f`, then sync. It panicked in round 1 without the fix (`vflush` → `vm_info_free`) and passed 9/9 with it. A write+sync variant also passed 9/9. - **Same recipe on the stock kernel** (no probe): it leaves a freed, dying, queued entry. On the final kernel, 3 runs: no panic, queue clean. - **Busy mount:** `check-mmap -d /mnt/nfs` then a plain umount went from Device busy 3/3 to success 3/3. - **Checks:** `check-fs`, `check-mmap` and `check-vm` pass on UFS and on `/mnt/nfs`. - **Full runs:** two all-category five-filesystem runs (seed 75): no panic, NFS plain unmount succeeds, every fsck clean. Their remaining FAILs are the known `fs rw` and `proc pty` failures, also seen on the stock kernel, plus a run-2 UFS phase that ran about 3x slow (likely host load). **Left open:** - **The negative count:** never caught at the exact decrement; a negative-count probe kernel wasn't built. - **Paths without the vnode lock:** those that reach a vm_info without it (`unmap_vnode` across `VOP_GETATTR`, `vmp_invalidate`, and `mapfs_sync` between its unlock and `vmp_get`) can still touch a freed entry if a forced unmount races them. - **`vmp_put` that drops `use_count` to 0:** `vm_info_free` can free the entry in the short gap before that `vmp_put` releases `vmp->lock`. - **Forced reclaim of an open file:** it still leaks its mfs window and object reference, as the stock kernel does.