[GH-ISSUE #72] net: unp_gc calls sorflush on non-socket files and freezes the guest #64

New issue

Closed

opened 2026-11-03 22:03:37 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:37 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/72

Symptom

vm/work/issue-67/gccycle.c sends an open regular file and a socketpair's own peer over SCM_RIGHTS, then closes everything. The guest freezes every time: unp_gc spins in sbflush (gdbstub, found while debugging #67).

Cause

unp_gc (bsd/kern/uipc_usrreq.c ~789-803) collects every unreachable file (f_count == f_msgcount, not FMARK) and calls sorflush((struct socket *)fp->f_data) on each one. For a vnode-backed file f_data is a vnode, not a socket, so sorflush/sbflush work on garbage. The mark phase also has a window: descriptors in flight can be marked or unmarked while unp_gc sleeps (M_WAITOK allocation) — check against 4.4BSD-Lite2/FreeBSD's version.

Fix direction

Only sorflush files whose f_type == DTYPE_SOCKET (as later BSDs do); closef still drops the extra reference for the rest.

Found by the #67 investigation.

Originally created by @raynorpat on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/72 ## Symptom `vm/work/issue-67/gccycle.c` sends an open regular file and a socketpair's own peer over SCM_RIGHTS, then closes everything. The guest freezes every time: `unp_gc` spins in `sbflush` (gdbstub, found while debugging #67). ## Cause `unp_gc` (`bsd/kern/uipc_usrreq.c` ~789-803) collects every unreachable file (`f_count == f_msgcount`, not FMARK) and calls `sorflush((struct socket *)fp->f_data)` on each one. For a vnode-backed file `f_data` is a vnode, not a socket, so `sorflush`/`sbflush` work on garbage. The mark phase also has a window: descriptors in flight can be marked or unmarked while `unp_gc` sleeps (M_WAITOK allocation) — check against 4.4BSD-Lite2/FreeBSD's version. ## Fix direction Only `sorflush` files whose `f_type == DTYPE_SOCKET` (as later BSDs do); `closef` still drops the extra reference for the rest. Found by the #67 investigation.

gitea-mirror closed this issue

2026-11-03 22:03:37 -07:00

gitea-mirror commented

2026-11-03 22:03:38 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 20, 2026):

Fixed by f2f657094 and 040bf8682, merged into ffs-perf in 4fb1150e2, plus dcaccc812.

f2f657094: unp_gc only sorflushes DTYPE_SOCKET files. The M_WAITOK allocation of the extra-reference array now happens before the mark phase, so it can't sleep between mark and sweep.
dcaccc812: because the array is sized from nfiles before the allocation sleeps, the sweep now stops collecting at the array's size. Any files left over wait for the next gc.
040bf8682, a second bug the reproducer exposed: unp_gc's closef(fp, 0) reaches vn_close with no process, and unmap_vnode (kern/mapfs.c) dereferenced p->p_cred. The closing process got SIGSEGV and leaked 2 file-table entries per close until the table filled up (ENFILE). vn_close now uses current_proc() when it's given none.

Verified on the kernel built from 040bf8682 (cksum 1995866079):

Before: gccycle 50 froze the guest. Sending only a regular file got SIGSEGV and leaked files.
After: gccycle 5000 and gcmode 1000 in all three modes pass, with no file-table growth.
stress-net fdpass, make check-net and make check-ipc pass.
soak -t 600 -j 4: no freeze or panic, net 6/6. Its vm pressure and ipc ool timeouts were also seen on earlier kernels, and two proc pty EIO failures are on paths these commits don't touch.
fsck matches the untouched base image.

dcaccc812 will be covered by the combined ffs-perf retest.

@raynorpat commented on GitHub (Oct 20, 2026): Fixed by f2f657094 and 040bf8682, merged into ffs-perf in 4fb1150e2, plus dcaccc812. - **f2f657094:** `unp_gc` only `sorflush`es `DTYPE_SOCKET` files. The `M_WAITOK` allocation of the extra-reference array now happens before the mark phase, so it can't sleep between mark and sweep. - **dcaccc812:** because the array is sized from `nfiles` before the allocation sleeps, the sweep now stops collecting at the array's size. Any files left over wait for the next gc. - **040bf8682, a second bug the reproducer exposed:** `unp_gc`'s `closef(fp, 0)` reaches `vn_close` with no process, and `unmap_vnode` (`kern/mapfs.c`) dereferenced `p->p_cred`. The closing process got SIGSEGV and leaked 2 file-table entries per close until the table filled up (ENFILE). `vn_close` now uses `current_proc()` when it's given none. **Verified** on the kernel built from 040bf8682 (cksum 1995866079): - **Before:** `gccycle 50` froze the guest. Sending only a regular file got SIGSEGV and leaked files. - **After:** `gccycle 5000` and `gcmode 1000` in all three modes pass, with no file-table growth. - `stress-net fdpass`, `make check-net` and `make check-ipc` pass. - `soak -t 600 -j 4`: no freeze or panic, net 6/6. Its `vm pressure` and `ipc ool` timeouts were also seen on earlier kernels, and two `proc pty` EIO failures are on paths these commits don't touch. - fsck matches the untouched base image. dcaccc812 will be covered by the combined ffs-perf retest.