[GH-ISSUE #42] kernel: forking past ~517 processes as root panics in ipc_task_init instead of returning EAGAIN #34

New issue

Closed

opened 2026-11-03 22:02:42 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:02:42 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/42

Symptom

On i386, a root process that forks until fork fails panics the kernel instead of getting EAGAIN:

System Panic: ipc_task_init

It happens around the 517th process. Root ignores the per-user RLIMIT_NPROC, so any root program that forks heavily can take the machine down: a runaway shell loop, a busy daemon, a parallel build.

Reproduce

The stress suite's supervision self-check (src/Commands/stress, ffs-perf branch) originally ran its forkbomb case as root. It forks paused children until fork fails, and that crashed the v2 guest's kernel (kernel-155.0.0-7, master + ffs fixes). The check now drops to uid -2 with RLIMIT_NPROC=64 so the dev guest survives. Run unprivileged under a limit, the same code gets fork: EAGAIN cleanly.

Minimal reproduction as root:

while :; do sleep 1000 & done

Cause

Two limits disagree:

BSD process limit: the i386 RELEASE config is medium (maxusers 32), so maxproc = 20 + 16 * 32 = 532 (src/kernel-7/conf/param.c:81-82). fork1() only refuses with EAGAIN once that limit is reached.
Mach IPC space limit: the IPC space zone is capped at SPACE_MAX = TASK_MAX + 5 = 517 (src/kernel-7/kern/mach_param.h:78, TASK_MAX 512), so ipc_space_create() runs out first.
The panic: ipc_task_init() (src/kernel-7/kern/ipc_tt.c:99-105) panics when ipc_space_create() or ipc_port_alloc_kernel() fails, instead of returning an error up through task_create to fork.

Possible fixes

Cap the process limit: clamp maxproc to fewer than TASK_MAX tasks, leaving room for kernel tasks, so fork refuses with EAGAIN before Mach runs out. This is a one-line config or param.c change, plus a check if maxproc is ever raised by sysctl.
Make task creation fail cleanly: have ipc_task_init / task_create return KERN_RESOURCE_SHORTAGE, and make fork1() unwind and return EAGAIN. That's more work, but it also covers running out of ports (ipc_port_alloc_kernel) and other zones.

Option 1 alone stops the panic for the common case. Option 2 is the robust fix.

Related: #40 and #41, also found by the stress suite.

Originally created by @raynorpat on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/42 ## Symptom On i386, a root process that forks until fork fails panics the kernel instead of getting `EAGAIN`: ``` System Panic: ipc_task_init ``` It happens around the 517th process. Root ignores the per-user `RLIMIT_NPROC`, so any root program that forks heavily can take the machine down: a runaway shell loop, a busy daemon, a parallel build. ## Reproduce The stress suite's supervision self-check (`src/Commands/stress`, ffs-perf branch) originally ran its `forkbomb` case as root. It forks paused children until fork fails, and that crashed the v2 guest's kernel (kernel-155.0.0-7, master + ffs fixes). The check now drops to uid -2 with `RLIMIT_NPROC=64` so the dev guest survives. Run unprivileged under a limit, the same code gets `fork: EAGAIN` cleanly. Minimal reproduction as root: ```sh while :; do sleep 1000 & done ``` ## Cause Two limits disagree: - **BSD process limit:** the i386 RELEASE config is `medium` (maxusers 32), so `maxproc = 20 + 16 * 32 = 532` (`src/kernel-7/conf/param.c:81-82`). `fork1()` only refuses with EAGAIN once that limit is reached. - **Mach IPC space limit:** the IPC space zone is capped at `SPACE_MAX = TASK_MAX + 5 = 517` (`src/kernel-7/kern/mach_param.h:78`, `TASK_MAX` 512), so `ipc_space_create()` runs out first. - **The panic:** `ipc_task_init()` (`src/kernel-7/kern/ipc_tt.c:99-105`) panics when `ipc_space_create()` or `ipc_port_alloc_kernel()` fails, instead of returning an error up through `task_create` to `fork`. ## Possible fixes 1. **Cap the process limit:** clamp `maxproc` to fewer than `TASK_MAX` tasks, leaving room for kernel tasks, so fork refuses with EAGAIN before Mach runs out. This is a one-line config or `param.c` change, plus a check if `maxproc` is ever raised by sysctl. 2. **Make task creation fail cleanly:** have `ipc_task_init` / `task_create` return `KERN_RESOURCE_SHORTAGE`, and make `fork1()` unwind and return EAGAIN. That's more work, but it also covers running out of ports (`ipc_port_alloc_kernel`) and other zones. Option 1 alone stops the panic for the common case. Option 2 is the robust fix. Related: #40 and #41, also found by the stress suite.

gitea-mirror

2026-11-03 22:02:42 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:02:44 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 14, 2026):

Fixed with option 2 in 389eb247d and 1fba3717a (local on master, not yet pushed): ipc_task_init now returns KERN_RESOURCE_SHORTAGE instead of panicking, task_create unwinds, procdup/cloneproc unwind the half-built proc, and fork1 returns EAGAIN.

Verified on the i386 guest (v2 image, 512 MB, kern.maxproc=532). The old kernel (kernel-155.0.0-7, Oct 10) panics with 'panic: ipc_task_init' under a root fork loop. With the rebuilt kernel the same root loop gets 'Cannot fork' and the guest keeps running (still up, ps works, no reboot).

@raynorpat commented on GitHub (Oct 14, 2026): Fixed with option 2 in 389eb247d and 1fba3717a (local on master, not yet pushed): ipc_task_init now returns KERN_RESOURCE_SHORTAGE instead of panicking, task_create unwinds, procdup/cloneproc unwind the half-built proc, and fork1 returns EAGAIN. Verified on the i386 guest (v2 image, 512 MB, kern.maxproc=532). The old kernel (kernel-155.0.0-7, Oct 10) panics with 'panic: ipc_task_init' under a root fork loop. With the rebuilt kernel the same root loop gets 'Cannot fork' and the guest keeps running (still up, ps works, no reboot).