[GH-ISSUE #84] kernel: exec of a file that a forced unmount reclaims faults in vnode_pager_setup on a NULL v_vm_info #78

New issue

Closed

opened 2026-11-03 22:04:13 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:04:13 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/84

Found by the #81 work: probe scenario S4 on branch issue-81-probe-fix, kernel cksum 370498376, log vm/work/issue-81/pfix-all-com2.log.

Symptom

A process execs a binary from an NFS (or other) mount while umount -f reclaims that vnode. The kernel traps:

unexpected kernel trap e eip 1ed3ac (_vm_pager_get+0x18), addr b000ff53

vnode_pager_setup reads vp->v_vm_info->pager with v_vm_info NULL.

Where

kern/mach_loader.c ~249 calls vnode_pager_setup(vp, FALSE, TRUE) on the exec'd vnode with no check that the vnode is still live.
vm/vnode_pager.c ~640-668 creates v_vm_info only under !MACH_NBC. With MACH_NBC it assumes it exists, and dereferences v_vm_info->pager both before and after vnode_pager_create, which can block.
A forced reclaim (vclean → vm_info_free) running while exec blocks in between leaves v_vm_info NULL. The vnode becomes a dead (VBAD) vnode.
Before #81, the earlier unmap_vnode in exec faulted first on the same NULL. The #81 fix now skips a reclaimed vnode there, so exec gets as far as the pager.

Expected

An exec racing a forced unmount fails cleanly (ENOENT, EBADF or LOAD_FAILURE → the exec returns an error) instead of faulting. The same applies to the other vnode_pager_setup callers (mmap of a file, swap files).

Originally created by @raynorpat on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/84 Found by the #81 work: probe scenario S4 on branch issue-81-probe-fix, kernel cksum 370498376, log `vm/work/issue-81/pfix-all-com2.log`. ## Symptom A process execs a binary from an NFS (or other) mount while `umount -f` reclaims that vnode. The kernel traps: ``` unexpected kernel trap e eip 1ed3ac (_vm_pager_get+0x18), addr b000ff53 ``` `vnode_pager_setup` reads `vp->v_vm_info->pager` with `v_vm_info` NULL. ## Where - `kern/mach_loader.c` ~249 calls `vnode_pager_setup(vp, FALSE, TRUE)` on the exec'd vnode with no check that the vnode is still live. - `vm/vnode_pager.c` ~640-668 creates `v_vm_info` only under `!MACH_NBC`. With MACH_NBC it assumes it exists, and dereferences `v_vm_info->pager` both before and after `vnode_pager_create`, which can block. - A forced reclaim (`vclean` → `vm_info_free`) running while exec blocks in between leaves `v_vm_info` NULL. The vnode becomes a dead (VBAD) vnode. - Before #81, the earlier `unmap_vnode` in exec faulted first on the same NULL. The #81 fix now skips a reclaimed vnode there, so exec gets as far as the pager. ## Expected An exec racing a forced unmount fails cleanly (ENOENT, EBADF or LOAD_FAILURE → the exec returns an error) instead of faulting. The same applies to the other `vnode_pager_setup` callers (mmap of a file, swap files).

gitea-mirror

2026-11-03 22:04:13 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:04:15 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 30, 2026):

Fixed by eba20b6c5, merged into ffs-perf in a2a72069f (on top of #81).

Root cause:

Under MACH_NBC, exec unlocks the vnode (kern_exec.c :258) but keeps using it. A umount -f can run vclean, which frees v_vm_info (vfs_subr.c :1254-1257).
parse_machfile then calls vnode_pager_setup (mach_loader.c :249), which reads v_vm_info->pager at entry, after vnode_pager_create (which can block in zalloc and VOP_GETATTR), and on return (vnode_pager.c :645/658/668).
parse_machfile also reads vnode_size again at :309.

Fix: vnode_pager_setup returns vm_pager_null when the vnode was reclaimed at entry or while it blocked. Every caller handles that:

exec / dyld (mach_loader.c): LOAD_FAILURE, so the user sees EBADEXEC (85). vnode_size is read once, right after setup.
mmap (kern_mman.c :342): EBADF.
map_fd (:791): KERN_INVALID_ARGUMENT; it frees its findspace allocation and guards the cred write.
map_vnode (mapfs.c): drops its pin on an orphaned vm_info.

Also:

vnode_pager_create frees a vstruct that was never published if the vnode was reclaimed during zalloc.
vnode_pagein null-checks v_vm_info, so a fault on a reclaimed mapped file gets SIGBUS instead of a panic.

Verified:

Before: on the #81 fix + probe kernel, the exec race trapped (_vm_pager_get+0x18).
After, fix + probe: 3/3 error 85, no trap.
Clean kernel (cksum 3234788009, same source as the merge apart from a comment): the race without the probe delay passed 5/5. rp.sh ×3 and rp2.sh ×3 rc=0. check-proc, check-mmap, check-vm pass on UFS and NFS, and the NFS plain unmount is clean.

Left open:

A vstruct is deliberately leaked if the reclaim lands inside VOP_GETATTR in vnode_pager_create.
vnode_pageout still reads v_vm_info->vnode_size unchecked (~850).
A reclaim while parse_machfile pages in its load commands wasn't reproduced.
mmap of an active swap file now fails instead of mapping a null pager.
The pager-file early return in vnode_pager_setup still skips unix_release.

@raynorpat commented on GitHub (Oct 30, 2026): Fixed by eba20b6c5, merged into ffs-perf in a2a72069f (on top of #81). **Root cause:** - Under MACH_NBC, exec unlocks the vnode (`kern_exec.c` :258) but keeps using it. A `umount -f` can run vclean, which frees `v_vm_info` (`vfs_subr.c` :1254-1257). - `parse_machfile` then calls `vnode_pager_setup` (`mach_loader.c` :249), which reads `v_vm_info->pager` at entry, after `vnode_pager_create` (which can block in `zalloc` and `VOP_GETATTR`), and on return (`vnode_pager.c` :645/658/668). - `parse_machfile` also reads `vnode_size` again at :309. **Fix:** `vnode_pager_setup` returns `vm_pager_null` when the vnode was reclaimed at entry or while it blocked. Every caller handles that: - **exec / dyld** (`mach_loader.c`): `LOAD_FAILURE`, so the user sees EBADEXEC (85). `vnode_size` is read once, right after setup. - **mmap** (`kern_mman.c` :342): EBADF. - **map_fd** (:791): KERN_INVALID_ARGUMENT; it frees its findspace allocation and guards the cred write. - **`map_vnode`** (`mapfs.c`): drops its pin on an orphaned vm_info. Also: - `vnode_pager_create` frees a vstruct that was never published if the vnode was reclaimed during `zalloc`. - `vnode_pagein` null-checks `v_vm_info`, so a fault on a reclaimed mapped file gets SIGBUS instead of a panic. **Verified:** - **Before:** on the #81 fix + probe kernel, the exec race trapped (`_vm_pager_get+0x18`). - **After, fix + probe:** 3/3 `error 85`, no trap. - **Clean kernel** (cksum 3234788009, same source as the merge apart from a comment): the race without the probe delay passed 5/5. `rp.sh` ×3 and `rp2.sh` ×3 rc=0. `check-proc`, `check-mmap`, `check-vm` pass on UFS and NFS, and the NFS plain unmount is clean. **Left open:** - A vstruct is deliberately leaked if the reclaim lands inside `VOP_GETATTR` in `vnode_pager_create`. - `vnode_pageout` still reads `v_vm_info->vnode_size` unchecked (~850). - A reclaim while `parse_machfile` pages in its load commands wasn't reproduced. - mmap of an active swap file now fails instead of mapping a null pager. - The pager-file early return in `vnode_pager_setup` still skips `unix_release`.

gitea-mirror referenced this issue

2026-11-03 22:04:16 -07:00

[GH-ISSUE #88] nfs: loopback NFS deadlocks when a client vnode is recycled (getnewvnode -> vgonel -> vinvalbuf -> nfs fsync) #79