nfs fsync) #79

New issue

Closed

opened 2026-11-03 22:04:16 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:04:16 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/88

Originally assigned to: @raynorpat on GitHub.

Found while testing #81 (pre-existing; reproduced on the stock kernel).

Symptom

vm/work/issue-81/ditest.c on a loopback NFS mount (export and mount on the same guest) writes 4×150 files of 24K, then recycles vnodes (ls -lR, cat across the system). It hangs on both the stock kernel and the #81 kernel, before verification.

Thread dumps (vm/work/issue-81/s1-hang-fw.txt, s1-hang-all.txt, s4-hang-all.txt stock; s3-hang-fw.txt fixed):

nfsd and stat: stuck in getnewvnode → vgonel → vinvalbuf+0x73, which is VOP_FSYNC: the NFS client buffer flush nfs_fsync / nfs_flush.
No mapfs frames.

Likely cause

An nfsd that needs a new vnode to serve a request recycles an NFS client vnode with dirty buffers. Flushing those buffers needs an nfsd to serve the writes, and over loopback that nfsd is this thread or one of its peers, blocked the same way.

This is the vnode-recycling counterpart of the mapfs self-deadlock fixed in #78 (e9e67317f).

Expected

Over loopback NFS, recycling never waits on its own server. Possible fixes:

getnewvnode skips dirty NFS client vnodes when the caller is an nfsd, or always when the free list offers a clean one;
or the recycle doesn't flush synchronously from server context.

Reproduce

ditest with NFS on the guest (-d /mnt/nfs, export via NetInfo as in vm/stress_fs.sh), 4×150 or more, with recycling. It also cuts the #81 data-integrity test short on NFS, so there are no NFS integrity counts yet.

Originally created by @raynorpat on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/88 Originally assigned to: @raynorpat on GitHub. Found while testing #81 (pre-existing; reproduced on the stock kernel). ## Symptom `vm/work/issue-81/ditest.c` on a loopback NFS mount (export and mount on the same guest) writes 4×150 files of 24K, then recycles vnodes (`ls -lR`, `cat` across the system). It hangs on both the stock kernel and the #81 kernel, before verification. Thread dumps (`vm/work/issue-81/s1-hang-fw.txt`, `s1-hang-all.txt`, `s4-hang-all.txt` stock; `s3-hang-fw.txt` fixed): - **nfsd and `stat`:** stuck in `getnewvnode` → `vgonel` → `vinvalbuf+0x73`, which is `VOP_FSYNC`: the NFS client buffer flush `nfs_fsync` / `nfs_flush`. - **No mapfs frames.** ## Likely cause An nfsd that needs a new vnode to serve a request recycles an NFS *client* vnode with dirty buffers. Flushing those buffers needs an nfsd to serve the writes, and over loopback that nfsd is this thread or one of its peers, blocked the same way. This is the vnode-recycling counterpart of the mapfs self-deadlock fixed in #78 (e9e67317f). ## Expected Over loopback NFS, recycling never waits on its own server. Possible fixes: - `getnewvnode` skips dirty NFS client vnodes when the caller is an nfsd, or always when the free list offers a clean one; - or the recycle doesn't flush synchronously from server context. ## Reproduce `ditest` with NFS on the guest (`-d /mnt/nfs`, export via NetInfo as in `vm/stress_fs.sh`), 4×150 or more, with recycling. It also cuts the #81 data-integrity test short on NFS, so there are no NFS integrity counts yet.

gitea-mirror

2026-11-03 22:04:16 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:04:19 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 31, 2026):

Fixed by 047687dbe, 84b136588 and 5e1ac8f9a, merged into ffs-perf in 550b7c1f1.

Deadlock 1 (base kernel, ditest NFS 4×150; thread dumps vm/work/issue-88/b1-hang-fw.txt):

nfsds in nfsrv_create/nfsrv_write → ffs_vget → getnewvnode → vgonel → vclean → vinvalbuf → nfs_flush, sleeping "nfsfsync" on a busy buffer (nfs_vnops.c :3066/:3102).
That buffer was queued by nfs_bwrite to an nfsiod, which is in nfs_writerpc waiting for a reply only an nfsd can send.
The remaining nfsds wait on the directory lock held inside nfsrv_create. All 4 nfsds are blocked.

Deadlock 2 (with the first fix alone): an nfsd → getnewvnode → vm_object_cache_steal → vm_object_terminate → vnode_pageout → nfs_write → nfs_asyncio, sleeping "nfsaio" (nfs_bio.c :1032), with every nfsiod waiting for nfsd replies.

Fixes:

047687dbe: getnewvnode prefers a free vnode with no NFS I/O pending (new vnode_nfs_pending: VT_NFS with dirty buffers, v_numoutput > 0, or a busy clean buffer). An nfsd (new P_NFSD, set around nfssvc_nfsd) never recycles a pending vnode. If none is clean it allocates a fresh one. Other callers fall back to the old take-the-head path.
84b136588: an nfsd never calls vm_object_cache_steal from getnewvnode.
5e1ac8f9a, a third bug that showed once the hangs were gone: NFS creates landed in the wrong directory (/mnt1, the export root, even hd0), and open got ENOTDIR. nfs_lookup read the cached vnode's v_id after VOP_ACCESS, an RPC that sleeps (nfs_vnops.c :956-962). So a vnode recycled meanwhile, even for a UFS file, passed the capability check. The base code had this too; nfsds now recycle clean client vnodes more often, which made it far more likely. v_id is now taken before the call.

No starvation or data loss:

An nfsd always gets a vnode and never sleeps on NFS I/O or object-cache pushes in getnewvnode.
The extra vnodes past desiredvnodes are bounded by the free vnodes with I/O pending.
A skipped vnode keeps its buffers, which update/sync (nfs_sync fsyncs every dirty vnode on the mount), buffer pressure or a later non-nfsd recycle still write.

Verified on the kernel built from 5e1ac8f9a (cksum 3847563290):

ditest NFS 4×150: base hangs → ok=600, bad=0, missing=0.
ditest NFS 4×800: base hangs → ok=3200, bad=0, missing=0, and a per-create server check found none misplaced.
ditest UFS 4×800: ok=3200.
Recipes and checks: rp.sh/rp2.sh rc=0. check-fs, check-mmap, check-vm pass on UFS and /mnt/nfs, and the plain NFS unmount is clean.
Harness --fs nfs, all categories, seed 75, twice: run_exit 0, PASS=98, FAIL=0, UNSUP=9, no unmount error.

Left open:

An nfsd that drops the last reference to a cached VM object can still trigger vm_object_cache_trim's synchronous NFS push (not seen to hang).
A non-nfsd caller holding an exported directory's UFS lock that falls back to flushing a dirty NFS vnode could in theory still cycle with nfsds waiting on that lock.
ffs_vget has no inode-hash recheck after sleeping, the race later BSDs fixed. It isn't needed here.
check-mmap -d /mnt/nfs leaves stress.mmap.*.map.* files in the export.

@raynorpat commented on GitHub (Oct 31, 2026): Fixed by 047687dbe, 84b136588 and 5e1ac8f9a, merged into ffs-perf in 550b7c1f1. **Deadlock 1** (base kernel, `ditest` NFS 4×150; thread dumps `vm/work/issue-88/b1-hang-fw.txt`): - nfsds in `nfsrv_create`/`nfsrv_write` → `ffs_vget` → `getnewvnode` → `vgonel` → `vclean` → `vinvalbuf` → `nfs_flush`, sleeping "nfsfsync" on a busy buffer (`nfs_vnops.c` :3066/:3102). - That buffer was queued by `nfs_bwrite` to an nfsiod, which is in `nfs_writerpc` waiting for a reply only an nfsd can send. - The remaining nfsds wait on the directory lock held inside `nfsrv_create`. All 4 nfsds are blocked. **Deadlock 2** (with the first fix alone): an nfsd → `getnewvnode` → `vm_object_cache_steal` → `vm_object_terminate` → `vnode_pageout` → `nfs_write` → `nfs_asyncio`, sleeping "nfsaio" (`nfs_bio.c` :1032), with every nfsiod waiting for nfsd replies. **Fixes:** - **047687dbe:** `getnewvnode` prefers a free vnode with no NFS I/O pending (new `vnode_nfs_pending`: VT_NFS with dirty buffers, `v_numoutput` > 0, or a busy clean buffer). An nfsd (new `P_NFSD`, set around `nfssvc_nfsd`) never recycles a pending vnode. If none is clean it allocates a fresh one. Other callers fall back to the old take-the-head path. - **84b136588:** an nfsd never calls `vm_object_cache_steal` from `getnewvnode`. - **5e1ac8f9a, a third bug that showed once the hangs were gone:** NFS creates landed in the wrong directory (`/mnt1`, the export root, even hd0), and open got ENOTDIR. `nfs_lookup` read the cached vnode's `v_id` after `VOP_ACCESS`, an RPC that sleeps (`nfs_vnops.c` :956-962). So a vnode recycled meanwhile, even for a UFS file, passed the capability check. The base code had this too; nfsds now recycle clean client vnodes more often, which made it far more likely. `v_id` is now taken before the call. **No starvation or data loss:** - An nfsd always gets a vnode and never sleeps on NFS I/O or object-cache pushes in `getnewvnode`. - The extra vnodes past `desiredvnodes` are bounded by the free vnodes with I/O pending. - A skipped vnode keeps its buffers, which update/sync (`nfs_sync` fsyncs every dirty vnode on the mount), buffer pressure or a later non-nfsd recycle still write. **Verified** on the kernel built from 5e1ac8f9a (cksum 3847563290): - **ditest NFS 4×150:** base hangs → ok=600, bad=0, missing=0. - **ditest NFS 4×800:** base hangs → ok=3200, bad=0, missing=0, and a per-create server check found none misplaced. - **ditest UFS 4×800:** ok=3200. - **Recipes and checks:** `rp.sh`/`rp2.sh` rc=0. `check-fs`, `check-mmap`, `check-vm` pass on UFS and `/mnt/nfs`, and the plain NFS unmount is clean. - **Harness `--fs nfs`, all categories, seed 75, twice:** run_exit 0, PASS=98, FAIL=0, UNSUP=9, no unmount error. **Left open:** - An nfsd that drops the last reference to a cached VM object can still trigger `vm_object_cache_trim`'s synchronous NFS push (not seen to hang). - A non-nfsd caller holding an exported directory's UFS lock that falls back to flushing a dirty NFS vnode could in theory still cycle with nfsds waiting on that lock. - `ffs_vget` has no inode-hash recheck after sleeping, the race later BSDs fixed. It isn't needed here. - `check-mmap -d /mnt/nfs` leaves `stress.mmap.*.map.*` files in the export.