[GH-ISSUE #67] net: stress-net fdpass (SCM_RIGHTS over Unix sockets) intermittently hangs until the watchdog #60

New issue

Closed

opened 2026-11-03 22:03:31 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:31 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/67

Symptom

stress-net fdpass hangs until its watchdog fires: STRESS net fdpass FAIL reason=timeout secs=600.

Phase 3 soak (2026-10-15): the test's controlling process (the leader) sat in its reap loop with no worker processes left and 0 CPU. 29 targeted re-runs did not reproduce it.
Phase 4 --fs acceptance, UFS run (2026-10-19, #60 kernel): it hung again. That guest's clock ran at half speed (#64), so the 600 s watchdog was about 20 real minutes, which makes this a real hang, not a slow test.
New lost-child check: the leader now ends with FAIL reason=lost-child when waitpid reports no children while it still counts live workers. It did not fire, so on this occurrence the workers were still alive, or waitpid was not returning ECHILD.

Reproduce

Intermittent. run 1 or run -a 1 with all categories, or a soak, on the dev guest. fdpass passes file descriptors over Unix-domain sockets with SCM_RIGHTS between worker processes (src/Commands/stress/net/net.c).

Evidence:

vm/work/stress-suite-evidence/phase3/final/fdpass-hang-note.md (phase 3: process state, analysis)
vm/work/ffsbench/runs/p4-accept/stress-0/net.out (phase 4)

Where to look

Kernel socket code:
- SCM_RIGHTS handling in bsd/kern/uipc_usrreq.c (unp_internalize/unp_externalize);
- a receiver blocked in soreceive waiting for a message whose rights could not be attached;
- a sender blocked in sosend on a full socket buffer whose reader has died.
Next step: catch the hang with a gdbstub thread dump of the workers' kernel stacks; earlier stress investigations' vm/work/issue59/fw*.txt show how. If it turns out to be a test bug (both ends waiting on each other by design), fix the test instead.

Found by the stress suite.

Originally created by @raynorpat on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/67 ## Symptom `stress-net fdpass` hangs until its watchdog fires: `STRESS net fdpass FAIL reason=timeout secs=600`. - **Phase 3 soak (2026-10-15):** the test's controlling process (the leader) sat in its reap loop with no worker processes left and 0 CPU. 29 targeted re-runs did not reproduce it. - **Phase 4 `--fs` acceptance, UFS run (2026-10-19, #60 kernel):** it hung again. That guest's clock ran at half speed (#64), so the 600 s watchdog was about 20 real minutes, which makes this a real hang, not a slow test. - **New `lost-child` check:** the leader now ends with `FAIL reason=lost-child` when `waitpid` reports no children while it still counts live workers. It did **not** fire, so on this occurrence the workers were still alive, or `waitpid` was not returning ECHILD. ## Reproduce Intermittent. `run 1` or `run -a 1` with all categories, or a soak, on the dev guest. `fdpass` passes file descriptors over Unix-domain sockets with `SCM_RIGHTS` between worker processes (`src/Commands/stress/net/net.c`). Evidence: - `vm/work/stress-suite-evidence/phase3/final/fdpass-hang-note.md` (phase 3: process state, analysis) - `vm/work/ffsbench/runs/p4-accept/stress-0/net.out` (phase 4) ## Where to look - **Kernel socket code:** - `SCM_RIGHTS` handling in `bsd/kern/uipc_usrreq.c` (`unp_internalize`/`unp_externalize`); - a receiver blocked in `soreceive` waiting for a message whose rights could not be attached; - a sender blocked in `sosend` on a full socket buffer whose reader has died. - **Next step:** catch the hang with a gdbstub thread dump of the workers' kernel stacks; earlier stress investigations' `vm/work/issue59/fw*.txt` show how. If it turns out to be a test bug (both ends waiting on each other by design), fix the test instead. Found by the stress suite.

gitea-mirror closed this issue

2026-11-03 22:03:32 -07:00

gitea-mirror commented

2026-11-03 22:03:33 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 20, 2026):

Fixed by 0a354086c (ffs: reclaim an old fifo's or device's vnode when ffs_valloc reuses its inode, so a new file does not keep fifo ops), merged into ffs-perf in 240f072ca.

Not #71: it reproduced on a kernel with the FSAVE workaround (ffs-perf 4fd6d07b6): fdpass beside stress-misc fifo in the same directory hung 3 times out of 3, within about a minute.

Not a socket bug. gdbstub showed:

The leader looping in select, its workers exiting (P_WEXIT, off allproc, so ps can't see them) but not yet zombies. So waitpid returned 0, and the lost-child check never fired.
Each worker stuck in an endless fault loop.
A hardware breakpoint found the first fault every time: a worker's write() to a newly created regular file went vn_write → ufsfifo_write → fifo_write → sosend(0xf000ff53), a garbage socket pointer. The new file's vnode was VREG, mode 0100600, but its v_op was ffs_fifoop_p.

Root cause:

ufs_inactive (ufs_inode.c ~117) frees the inode in the cylinder-group bitmap before it recycles the vnode (~130).
If ffs_valloc (ffs_alloc.c ~669) reallocates that inode in between, VFS_VGET returns the old vnode from the inode hash.
When the old file was a fifo, that vnode keeps fifo ops, and its v_un (fifo info) is reused as vm_info, so the next write goes through fifo_write with a garbage socket.
trap.c (~255-272) treats a kernel fault below KERNEL_LINEAR_BASE like a user fault, sending the thread back to user mode with kernel registers. That is the endless loop.

Fix: if the vnode ffs_valloc gets back doesn't have ffs_vnodeop_p, it is released with vput (its i_mode is 0, so ufs_inactive reclaims it), and ffs_valloc fetches a fresh vnode. Device vnodes are covered by the same check.

Verified on the fixed kernel (cksum 606865771):

fdpass beside fifo: 342 fdpass and 115 fifo runs over 60 min, all PASS.
Soak plus an fdpass loop: 138 fdpass runs and 5 soaks, no hang.
A fresh UFS scratch volume: 83 fdpass and 28 fifo runs, then fsck -n clean.
make check-net, check-misc and check-fs pass.

Found along the way, not fixed here:

unp_gc in uipc_usrreq.c can freeze the guest. A socketpair carrying an open file plus its own peer spins forever in sbflush.
The trap.c behaviour above turns any kernel NULL dereference in a syscall into an unkillable, invisible loop.
ext2fs's valloc wasn't checked for the same inode-reuse problem.

@raynorpat commented on GitHub (Oct 20, 2026): Fixed by 0a354086c (ffs: reclaim an old fifo's or device's vnode when ffs_valloc reuses its inode, so a new file does not keep fifo ops), merged into ffs-perf in 240f072ca. **Not #71:** it reproduced on a kernel with the FSAVE workaround (ffs-perf 4fd6d07b6): fdpass beside `stress-misc fifo` in the same directory hung 3 times out of 3, within about a minute. **Not a socket bug.** gdbstub showed: - The leader looping in `select`, its workers exiting (P_WEXIT, off `allproc`, so `ps` can't see them) but not yet zombies. So `waitpid` returned 0, and the lost-child check never fired. - Each worker stuck in an endless fault loop. - A hardware breakpoint found the first fault every time: a worker's `write()` to a newly created regular file went `vn_write` → `ufsfifo_write` → `fifo_write` → `sosend(0xf000ff53)`, a garbage socket pointer. The new file's vnode was VREG, mode 0100600, but its `v_op` was `ffs_fifoop_p`. **Root cause:** - `ufs_inactive` (`ufs_inode.c` ~117) frees the inode in the cylinder-group bitmap before it recycles the vnode (~130). - If `ffs_valloc` (`ffs_alloc.c` ~669) reallocates that inode in between, `VFS_VGET` returns the old vnode from the inode hash. - When the old file was a fifo, that vnode keeps fifo ops, and its `v_un` (fifo info) is reused as `vm_info`, so the next write goes through `fifo_write` with a garbage socket. - `trap.c` (~255-272) treats a kernel fault below `KERNEL_LINEAR_BASE` like a user fault, sending the thread back to user mode with kernel registers. That is the endless loop. **Fix:** if the vnode `ffs_valloc` gets back doesn't have `ffs_vnodeop_p`, it is released with `vput` (its `i_mode` is 0, so `ufs_inactive` reclaims it), and `ffs_valloc` fetches a fresh vnode. Device vnodes are covered by the same check. **Verified** on the fixed kernel (cksum 606865771): - fdpass beside fifo: 342 fdpass and 115 fifo runs over 60 min, all PASS. - Soak plus an fdpass loop: 138 fdpass runs and 5 soaks, no hang. - A fresh UFS scratch volume: 83 fdpass and 28 fifo runs, then `fsck -n` clean. - `make check-net`, `check-misc` and `check-fs` pass. **Found along the way, not fixed here:** - `unp_gc` in `uipc_usrreq.c` can freeze the guest. A socketpair carrying an open file plus its own peer spins forever in `sbflush`. - The `trap.c` behaviour above turns any kernel NULL dereference in a syscall into an unkillable, invisible loop. - ext2fs's valloc wasn't checked for the same inode-reuse problem.

gitea-mirror referenced this issue

2026-11-03 22:03:34 -07:00

[GH-ISSUE #68] fork: a child forked from a cthreads process intermittently dies with SIGSEGV (stress-thread fork) #63