[GH-ISSUE #87] kernel: msync returns EIO when another process is writing the same file (vnode_pageout gives up on the vnode lock) #81

New issue

Closed

opened 2026-11-03 22:04:16 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:04:16 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/87

Originally assigned to: @raynorpat on GitHub.

Found while testing #81 (pre-existing; reproduced on the stock kernel).

Symptom

vm/work/issue-81/msx.c: one process loops msync on a dirty 64K shared mapping of a UFS file while a second process write()s 1 MB chunks to the same file, for 60 s.

Stock (3d017f064): 246 of 297 and 249 of 281 msync calls failed with EIO.
With #81: 250 of 291.

The stress suite's mmap sync hit it once as msync: EIO under load (runs/i81).

Cause

vnode_pageout (vm/vnode_pager.c ~915-930 on ffs-perf 23c1dd5f6) takes the vnode lock with LK_NOWAIT, with VNODE_LOCK_RETRY_COUNT 1 and VNODE_LOCK_RETRY_TICKS 2. If a writer holds the lock for longer than about 2 ticks, the pageout returns PAGER_ERROR, and msync (vm_synchronize / vm_msync) reports EIO.

The no-wait lock is probably there so the pageout daemon never blocks on a vnode, which can deadlock (for example against a thread holding the vnode lock and waiting for free pages).

Expected

A synchronous msync (MS_SYNC) on a healthy file never fails because another process is writing it: it waits for the lock, or retries until it gets it. The pageout daemon may keep its no-wait behaviour. The fix must tell the two callers apart (for example a flag on the pageout request, or a caller-context check) and must not bring back the pageout-daemon deadlock.

Originally created by @raynorpat on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/87 Originally assigned to: @raynorpat on GitHub. Found while testing #81 (pre-existing; reproduced on the stock kernel). ## Symptom `vm/work/issue-81/msx.c`: one process loops `msync` on a dirty 64K shared mapping of a UFS file while a second process `write()`s 1 MB chunks to the same file, for 60 s. - Stock (3d017f064): 246 of 297 and 249 of 281 msync calls failed with EIO. - With #81: 250 of 291. The stress suite's `mmap sync` hit it once as `msync: EIO` under load (`runs/i81`). ## Cause `vnode_pageout` (`vm/vnode_pager.c` ~915-930 on ffs-perf 23c1dd5f6) takes the vnode lock with LK_NOWAIT, with `VNODE_LOCK_RETRY_COUNT 1` and `VNODE_LOCK_RETRY_TICKS 2`. If a writer holds the lock for longer than about 2 ticks, the pageout returns PAGER_ERROR, and `msync` (`vm_synchronize` / `vm_msync`) reports EIO. The no-wait lock is probably there so the pageout daemon never blocks on a vnode, which can deadlock (for example against a thread holding the vnode lock and waiting for free pages). ## Expected A synchronous `msync` (MS_SYNC) on a healthy file never fails because another process is writing it: it waits for the lock, or retries until it gets it. The pageout daemon may keep its no-wait behaviour. The fix must tell the two callers apart (for example a flag on the pageout request, or a caller-context check) and must not bring back the pageout-daemon deadlock.

gitea-mirror

2026-11-03 22:04:16 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:04:19 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 31, 2026):

Fixed by f3202b49e, merged into ffs-perf in 1f8535d41.

Why the lock is no-wait. vnode_pageout (vm/vnode_pager.c :916-939) has taken it with LK_NOWAIT since the Darwin 0.3 import. Making it wait would deadlock two ways:

The pageout daemon (vm_pageout.c :291) is the only thread that frees memory, and the vnode-lock holder can be waiting for free pages.
Busy pages. Every caller marks the page busy before the pageout. A lock holder that touches that page (mapfs_io's uiomove, mapfs.c :1514, via vm_fault.c :269-282; vmp_push_range; vno_flush) waits on the busy bit and never releases the lock.

So a "wait" flag on the request would deadlock the reproducer itself.

Fix: by lock ownership. vnode_pageout stays LK_NOWAIT|LK_CANRECURSE. A caller that may sleep takes the vnode lock itself before any page is busy, so the pageout's lock is a recursion and can't fail (lockmgr checks the thread, kern_lock.c :341).

msync (vm/vm_synchronize.c map_push_range): for a file-backed object (not device, not swap file) it takes an object reference, drops the map read lock, takes the vnode lock (LK_RETRY|LK_CANRECURSE), pushes, unlocks, relocks the map and goes on after the range. The vnode lock is taken before the map lock and before any busy page, the same order a writer uses.
vm_pageout_page skips pages being paged in (nfspagereq), whose pagein may wait for the lock the pusher holds.
fsync (vfs_syscalls.c :2018) runs mapfs_fsync under the vnode lock it already takes. Its contended pushes used to fail silently.

The pageout daemon, vm_object_terminate, mapfs_sync and the VXLOCK paths don't pre-lock, so they keep no-wait.

Verified on the kernel built from f3202b49e (cksum 32422712):

msx, msync EIO:
- UFS: base 220/256 → 0/204
- NFS: 0/1553 → 0/1539
- UFS during 128 MB pressure: 0/300
ditest UFS: ok=3200, fsck clean.
Recipes: stress-mmap sync 5/5. The rp, rp2, s4x and mx recipes: every umount -f returns 0, no hang.
Checks: check-proc, check-mmap, check-vm, check-fs pass on UFS and NFS.
Soak 600 s: one proc pty EIO, the known failure from the final retests.
stress-vm pressure: passes at -m 192m on 128 MB. At 640m it hit the time limit with the guest responsive, the known #89 limit.

Left open:

Sub-map entries still wait for the vnode lock under the parent map's read lock.
mapfs_sync (update) and vm_object_terminate still give up on a busy vnode, and mapfs_sync stops its scan on the first error.
fsync ignores mapfs_fsync's return value.

@raynorpat commented on GitHub (Oct 31, 2026): Fixed by f3202b49e, merged into ffs-perf in 1f8535d41. **Why the lock is no-wait.** `vnode_pageout` (`vm/vnode_pager.c` :916-939) has taken it with LK_NOWAIT since the Darwin 0.3 import. Making it wait would deadlock two ways: 1. **The pageout daemon** (`vm_pageout.c` :291) is the only thread that frees memory, and the vnode-lock holder can be waiting for free pages. 2. **Busy pages.** Every caller marks the page busy before the pageout. A lock holder that touches that page (mapfs_io's `uiomove`, `mapfs.c` :1514, via `vm_fault.c` :269-282; `vmp_push_range`; `vno_flush`) waits on the busy bit and never releases the lock. So a "wait" flag on the request would deadlock the reproducer itself. **Fix: by lock ownership.** `vnode_pageout` stays LK_NOWAIT|LK_CANRECURSE. A caller that may sleep takes the vnode lock itself before any page is busy, so the pageout's lock is a recursion and can't fail (lockmgr checks the thread, `kern_lock.c` :341). - **msync** (`vm/vm_synchronize.c` `map_push_range`): for a file-backed object (not device, not swap file) it takes an object reference, drops the map read lock, takes the vnode lock (LK_RETRY|LK_CANRECURSE), pushes, unlocks, relocks the map and goes on after the range. The vnode lock is taken before the map lock and before any busy page, the same order a writer uses. - **`vm_pageout_page`** skips pages being paged in (`nfspagereq`), whose pagein may wait for the lock the pusher holds. - **fsync** (`vfs_syscalls.c` :2018) runs `mapfs_fsync` under the vnode lock it already takes. Its contended pushes used to fail silently. The pageout daemon, `vm_object_terminate`, `mapfs_sync` and the VXLOCK paths don't pre-lock, so they keep no-wait. **Verified** on the kernel built from f3202b49e (cksum 32422712): - **msx, msync EIO:** - UFS: base 220/256 → **0/204** - NFS: 0/1553 → 0/1539 - UFS during 128 MB pressure: 0/300 - **ditest UFS:** ok=3200, fsck clean. - **Recipes:** `stress-mmap sync` 5/5. The rp, rp2, s4x and mx recipes: every `umount -f` returns 0, no hang. - **Checks:** `check-proc`, `check-mmap`, `check-vm`, `check-fs` pass on UFS and NFS. - **Soak 600 s:** one `proc pty` EIO, the known failure from the final retests. - **`stress-vm pressure`:** passes at `-m 192m` on 128 MB. At 640m it hit the time limit with the guest responsive, the known #89 limit. **Left open:** - Sub-map entries still wait for the vnode lock under the parent map's read lock. - `mapfs_sync` (update) and `vm_object_terminate` still give up on a busy vnode, and `mapfs_sync` stops its scan on the first error. - fsync ignores `mapfs_fsync`'s return value.