mirror of
https://github.com/RhapsodiOS/RhapsodiOS.git
synced 2026-11-03 22:01:40 -07:00
[GH-ISSUE #45] kernel: concurrent rename/rmdir/unlink panics UFS (rename: lost dir entry, free vnode isn't, vget: v_usecount) #39
Labels
No labels
bug
documentation
enhancement
enhancement
good first issue
help wanted
help wanted
pull-request
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
github-starred/RhapsodiOS#39
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @raynorpat on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/45
Symptom
Concurrent namespace churn on one UFS directory tree panics the kernel within a handful of runs. Three different panics have been seen:
rename: lost dir entrybsd/ufs/ufs/ufs_vnops.c:1215or:1231(ufs_rename)free vnode isn't: v_type = 8, v_usecount = -1bsd/vfs/vfs_subr.c:455(getnewvnode)7ed3069ab)vget: v_usecountbsd/vfs/vfs_subr.c:876v_type = 8isVBAD: a vnode that was reclaimed or revoked, sitting on the free list with a use count of -1, so something calledvreleonce too often. Thevgetpanic is the same reference count gone wrong from the other side. That it happens even with directory renames removed suggests the refcount bug is not only in the directory-rename path.Reproduce
stress-fs treefrom the stress suite (src/Commands/stress, ffs-perf branch). It runs 4 processes doing randommkdir/rmdir/create/unlink/rename(of files, subdirectories and top-level directories) in a shared 3-level treed<0-7>/d<0-7>/f<0-7>under/tmp, then walks it:-cpu pentium), uniprocessor, 128 MB. The problem must therefore be in sleep/wakeup interleaving, not SMP.Where to look
ufs_rename(ufs_vnops.c~1100-1300):relookup()after the target is entered. The code assumes a directory source cannot disappear in between, because its link count blocksrmdirandIRENAMEblocks a concurrent rename.IRENAMEis not honoured on every path (e.g. a rename of the parent or of a sibling that moves the source), or a concurrent rename/rmdir slips in whileufs_renamesleeps inrelookupor a block read.genfs_rename.vrele/vputcalls on the error and "source vanished" branches, for examplevrele(ap->a_fvp)atufs_vnops.c:1216.ufs_rmdir/ufs_removeracing withvgetof a vnode being reclaimed (theVXLOCKpath atvfs_subr.c:863-868).vrelewould produce exactlyv_usecount = -1on aVBADvnode.msdosfs(msdosfs_vnops.c:1183/1204) andext2fs(ext2fs_vnops.c:808/824) carry the samelost dir entrypanics and probably the same race.Suggested fix direction
ENOENTinstead of panicking, with correct releases on every exit path.vrele/vputinufs_rename,ufs_rmdir,ufs_removeandrelookupagainst its reference.stress-fs -n 4 treein a loop of 40 seeds should run clean afterwards.Impact
Any concurrent build or
make clean, mail spool, or package install that renames and removes directories while others do the same can panic the system. This may also be behind some of the unexplained build-time crashes seen earlier.Related: #41, #42, #43, also found by the stress suite.
@raynorpat commented on GitHub (Oct 14, 2026):
Fixed on branch
fs-races(commits9db81ac41and690848e2b), not yet merged to master.Two causes, both in
ufs_rename:rmdiror rename in that window left the source entry missing or pointing at a different inode, andufs_renamehitpanic("rename: lost dir entry"). For directory renames it now re-checks the source name before changing anything and returns ENOENT. (9db81ac41)relookup()doesvput(dvp)when its lookup fails, but the rename syscall still releases the SAVESTART start directory afterwards. A failed relookup inufs_renametherefore dropped one reference too many: use count -1 on a vnode on the free list, which is thefree vnode isn't: v_type = 8, v_usecount = -1andvget: v_usecountpanics.ufs_renamenow calls a smallufs_relookup()wrapper that takes an extra reference first, the same pattern asunion_relookup. (690848e2b)Tested on an i386 QEMU guest (128 MB, snapshot of the v2 image). With only the first commit,
stress-fs -n 4 treestill panicked on every seed 1..10. With both commits, seeds 1..200 all pass with no panic, and-n 8 -i 3000 treepasses on seeds 101..106.Not done:
relookuppattern and probably the same use-count bug. I did not change them.fvp == tvpbranch ofufs_renamewould dereference NULL if itsrelookupfails.vgetwith directory renames removed from the test) was not run separately.Closing as fixed for UFS; the other filesystems should get their own issue.
@raynorpat commented on GitHub (Oct 14, 2026):
Follow-up on the items listed as not done above. Two more commits on branch
fs-races(not yet merged to master):b2381210b: ext2fs and msdosfs.ext2fs_renameis a copy of the UFS code, so it gets the same two fixes: re-check the source name for directory renames before changing anything, and keep the start-directory reference when arelookupfails.msdosfs_renamegets the same source re-check, done before the target entry is removed, and the extra reference at its target-directoryrelookup. Its source-directoryrelookupalready balances its references, so I left it alone. The reference-keeping helper is now a sharedrelookup_startdir()invfs_lookup.c, which UFS uses as well.6f587a3f9: thefvp == tvpbranch ofufs_renameandext2fs_renamecalledVOP_REMOVEwith the result of an uncheckedrelookup. If the source name was already gone that was a NULL dereference; it now returns 0.Status: these two commits have not been built or run yet. A kernel build and tree-stress run on UFS, msdosfs and ext2fs volumes is in progress, and I'll post the results here.
@raynorpat commented on GitHub (Oct 14, 2026):
Test results for the follow-up commits.
Kernel k3 is
fs-racesat89e39d0b2, which has the ext2fs/msdosfs changes, the NULL-deref fix, and a rework of the msdosfs re-check that unlocks the target before relooking up the source. Each run uses a fresh snapshot guest at 128 MB, runningstress-fs -n 4 -s <seed> tree.ext2fs_rename: lost dir entrythree times,free vnode isn't: v_type = 8, v_usecount = -1once).-n 1passes 60 of 60. But with-n 4it still hangs intermittently, usually within 10 to 50 seeds. The CPU spins in the lockmgr interlock invfs_busy, reached from rename → namei → lookup crossing the ext2 mount point, and themnt_lockword holds a pointer value. Not root-caused yet, and not yet known whether it predates these changes.PDIRUNLOCK,NDF_ONLY_PNBUF,msdos_vfsops, a 2-argumentVOP_STRATEGY). So the msdosfs rename change has not run.The UFS part is merged into
ffs-perf(mergee2baceb11), not master:76c273754,9db81ac41,690848e2b, plus508cac1b4, which carries the sharedrelookup_startdir()and the UFS half of the NULL-deref fix. These UFS/VFS files are identical to what k3 ran. The ext2fs and msdosfs commits stay onfs-raceswhile the ext2fs hang and the msdosfs build are worked on.@raynorpat commented on GitHub (Oct 15, 2026):
The ext2fs hang is fixed, and it was not a rename bug.
Root cause.
struct vnodekeepsv_mountedhere(for directories) andv_vm_info(for regular files) in the same union. When an unlinked ext2 file goes inactive,ext2fs_inactivefrees the inode number before it recycles the vnode. ext2 hands out the lowest free inode, so a concurrent mkdir can get that number back.ext2fs_vgetthen finds the old file's vnode, still in the inode hash, andext2fs_mkdirmakes it a directory. The old file'svm_infopointer is now read asv_mountedhere, so the next lookup through that directory callsvfs_busyon avm_info. The word at the mount lock's interlock offset isvm_info->vnode, a pointer, solockmgrspins forever. gdbstub dumps show exactly this: themppassed tovfs_busyis the directory's own formervm_info.This was already in the code, not caused by the rename changes. The stock kernel's rename panics fired first and hid it.
Fix.
ext2fs_vallocnow frees a reused vnode'svm_infoand resetsv_type, the same cleanupvcleandoes (9c0fd8de6).Results on the fixed kernel,
stress-fs -n 4 treeon a 128 MB guest:The ext2fs commits are merged into
ffs-perf(mergeab9388565); the ext2fs sources there are the ones that were tested. msdosfs is still being worked on, on its own branch.UFS has the same latent pattern:
ufs_inactivealso frees the inode before recycling the vnode. It probably does not show up because FFS rarely hands a just-freed inode straight back. It is not fixed or tested yet.@raynorpat commented on GitHub (Oct 15, 2026):
msdosfs is done, and the rename fix is confirmed there too.
msdosfs had never been compiled into the i386 kernel. Five commits make it build and work:
PDIRUNLOCK,NDFREE,vn_isdisk,malloc/free, the 2-argumentVOP_STRATEGY,vtruncbufand others) replaced with this kernel's equivalents.extendfilekeyed its zeroed buffers by sector instead of by cluster. Truncate followed by extend could return old bytes, or a late zero-write could overwrite real data.Tested on a 128 MB guest. Smoke tests pass on FAT16 and FAT32 (create, write, read, long names, mkdir and rmdir, renames, truncate and extend, remount).
fsck_msdoswas clean after every stress seed.stress-fs tree-n 1, seeds 1..40-n 4, seeds 1..200-n 4, seeds 1..40-n 4, seeds 1..40rename: lost dir entryMerged into
ffs-perf; the msdosfs sources there are the ones that were tested. msdosfs is not yet turned on in the shipped i386 kernel config. Not tested: executing or mmap-ing files on msdosfs, since msdosfs never sets up per-file VM info and its mmap entry returns an error.@raynorpat commented on GitHub (Oct 15, 2026):
Follow-up from the stress suite phase 3 (
stress-fs rename/link, branchstress-phase3):renametest (workersrm -rfthe shared tree while others still rename in it) panicked the v2 image's own kernel (pre-fix, built 2026-10-10) withfree vnode isn't: v_type = 8, v_usecount = -1— the panic listed above.9db81ac41,690848e2b,b2381210band89e39d0b2): 10 runs plus ~560 restarts over 10×60 s of rm -rf againststress-fs rename, andtreeseeds 1-20 plusrename linkseeds 1-10 on both the root fs and a UFS scratch disk — no panic, no hang, fsck clean.89e39d0b2), #55 (msdosfs mmap size loss).