[GH-ISSUE #49] fork from a multithreaded (cthreads) process: child sometimes hangs in malloc under load #40

New issue

Closed

opened 2026-11-03 22:03:05 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:05 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/49

Symptom

A child forked from a multithreaded (cthreads) process sometimes hangs. It never completes a plain malloc(1 MB) + fill + check + _exit(0). The parent waits 10 s and gives up.

Seen in the stress suite's stress-thread fork test (ffs-perf branch, src/Commands/stress/thread/thread.c). The setup:

the worker starts 4 cthreads that loop locking and unlocking a shared mutex_t and incrementing counters;
the main thread fork()s while they run;
the child does malloc, fill, check, then _exit(0).

STRESS thread fork FAIL reason=child-hang worker=1 iter=0 seed=487971239

Kernel: the v2 image's kernel (kernel-155.0.0-7, master + ffs fixes).
Guest: i386 QEMU TCG, 128 MB.
When it happens: only in the concurrent pass of thread/runtests, with all six thread tests running at once. It passed when run alone. No console or serial message accompanied it.

What the source already does (so the obvious cause is not it)

The fork path is meant to be safe for exactly this case:

The fork stub: fork() (src/Libc-1/sys.subproj/i386.subproj/fork.s) calls _cthread_fork_prepare and __dyld_fork_prepare before the syscall.
The child side: in the child it calls _cthread_fork_child, __dyld_fork_child and _dyld_fork_child_final.
The cthreads hooks: _cthread_fork_prepare (threads.subproj/cthreads.c:362) takes cthread_lock, cthread_free_lock, and malloc's lock through _malloc_fork_prepare (gen.subproj/malloc.c:2642). _cthread_fork_child (cthreads.c:387) releases them and resets cthreads state.

So the plain "malloc lock held by another thread at fork time" explanation should not apply. The real cause is not known yet.

Candidates to check

A lock outside the fork hooks:
- a spin_lock or mutex_t held by a parent thread at fork time that the child later needs and that isn't in _cthread_fork_prepare (e.g. mig_support.c reply-port state, stack.c, lookupd/_lu_port, the zone allocator's region locks if nxzone* uses its own);
- the dyld lock path after the phase 2 dyld change (signals blocked while the dyld lock is held, 733bbd587). Blocked signals are inherited across fork, so check whether the child restores the mask.
The kernel side:
- fork of a task with several Mach threads, and whether the child's single thread can deadlock with a copied but never-released kernel-side lock;
- a COW page-fault stall on the child's first touches under memory pressure.
Simply slow: the child may just be very slow under heavy TCG load. A longer bound would tell a hang from slowness.

A triage run on the stress suite is reproducing this in isolation and under load. It will post its findings here: where the child is blocked, and the call chain if it can be captured.

Reproduce

cd /usr/local/stress/thread        # or the built tree
sh runtests -n 4 1                 # concurrent pass hits it intermittently
./stress-thread -n 4 -i 200 fork   # alone, larger loop

Related: #42 (fork exhaustion panic), and the dyld signal-lock fix (733bbd587).

Originally created by @raynorpat on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/49 ## Symptom A child forked from a multithreaded (cthreads) process sometimes hangs. It never completes a plain `malloc(1 MB)` + fill + check + `_exit(0)`. The parent waits 10 s and gives up. Seen in the stress suite's `stress-thread fork` test (ffs-perf branch, `src/Commands/stress/thread/thread.c`). The setup: - the worker starts 4 cthreads that loop locking and unlocking a shared `mutex_t` and incrementing counters; - the main thread `fork()`s while they run; - the child does `malloc`, fill, check, then `_exit(0)`. ``` STRESS thread fork FAIL reason=child-hang worker=1 iter=0 seed=487971239 ``` - **Kernel:** the v2 image's kernel (kernel-155.0.0-7, master + ffs fixes). - **Guest:** i386 QEMU TCG, 128 MB. - **When it happens:** only in the concurrent pass of `thread/runtests`, with all six thread tests running at once. It passed when run alone. No console or serial message accompanied it. ## What the source already does (so the obvious cause is not it) The fork path is meant to be safe for exactly this case: - **The fork stub:** `fork()` (`src/Libc-1/sys.subproj/i386.subproj/fork.s`) calls `_cthread_fork_prepare` and `__dyld_fork_prepare` before the syscall. - **The child side:** in the child it calls `_cthread_fork_child`, `__dyld_fork_child` and `_dyld_fork_child_final`. - **The cthreads hooks:** `_cthread_fork_prepare` (`threads.subproj/cthreads.c:362`) takes `cthread_lock`, `cthread_free_lock`, and malloc's lock through `_malloc_fork_prepare` (`gen.subproj/malloc.c:2642`). `_cthread_fork_child` (`cthreads.c:387`) releases them and resets cthreads state. So the plain "malloc lock held by another thread at fork time" explanation should not apply. The real cause is not known yet. ## Candidates to check - **A lock outside the fork hooks:** - a `spin_lock` or `mutex_t` held by a parent thread at fork time that the child later needs and that isn't in `_cthread_fork_prepare` (e.g. `mig_support.c` reply-port state, `stack.c`, `lookupd`/`_lu_port`, the zone allocator's region locks if `nxzone*` uses its own); - the dyld lock path after the phase 2 dyld change (signals blocked while the dyld lock is held, 733bbd587). Blocked signals are inherited across fork, so check whether the child restores the mask. - **The kernel side:** - `fork` of a task with several Mach threads, and whether the child's single thread can deadlock with a copied but never-released kernel-side lock; - a COW page-fault stall on the child's first touches under memory pressure. - **Simply slow:** the child may just be very slow under heavy TCG load. A longer bound would tell a hang from slowness. A triage run on the stress suite is reproducing this in isolation and under load. It will post its findings here: where the child is blocked, and the call chain if it can be captured. ## Reproduce ``` cd /usr/local/stress/thread # or the built tree sh runtests -n 4 1 # concurrent pass hits it intermittently ./stress-thread -n 4 -i 200 fork # alone, larger loop ``` Related: #42 (fork exhaustion panic), and the dyld signal-lock fix (733bbd587).

gitea-mirror

2026-11-03 22:03:05 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:03:07 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 14, 2026):

Triage result from the stress-suite run: not a hang, and not a Libc or kernel bug as far as the evidence goes.

fork.s calls the cthreads fork hooks, which take and reset malloc's lock (threads.subproj/cthreads.c:362-400, gen.subproj/malloc.c:2642). malloc is fork-safe here.
The child was slow, not stuck. Every child completed its 1 MB malloc + fill + check + _exit(0), taking 6-15 s under the concurrent-test load and 7.2 s even when run alone, against the test's 10 s bound.
The test now uses its file-wide 60 s limit (stress commit 97199346c on ffs-perf). A rerun with the same seed under the same load passed.

The one thing still worth a look is why a fork plus 1 MB touch takes about 7 s in a 4-thread process on the TCG guest. That is a performance question, not a correctness bug. Recommend closing this issue, and opening a performance one if that timing matters.

@raynorpat commented on GitHub (Oct 14, 2026): Triage result from the stress-suite run: **not a hang, and not a Libc or kernel bug as far as the evidence goes.** - `fork.s` calls the cthreads fork hooks, which take and reset malloc's lock (`threads.subproj/cthreads.c:362-400`, `gen.subproj/malloc.c:2642`). malloc is fork-safe here. - The child was **slow, not stuck**. Every child completed its 1 MB malloc + fill + check + `_exit(0)`, taking 6-15 s under the concurrent-test load and 7.2 s even when run alone, against the test's 10 s bound. - The test now uses its file-wide 60 s limit (stress commit 97199346c on ffs-perf). A rerun with the same seed under the same load passed. The one thing still worth a look is why a fork plus 1 MB touch takes about 7 s in a 4-thread process on the TCG guest. That is a performance question, not a correctness bug. Recommend closing this issue, and opening a performance one if that timing matters.

gitea-mirror referenced this issue

2026-11-03 22:03:33 -07:00

[GH-ISSUE #69] kernel: i386 8254 system timestamp loses or gains whole ticks when IRQ 0 deliveries merge under CPU hold-off #61

gitea-mirror referenced this issue

2026-11-03 22:03:35 -07:00

[GH-ISSUE #69] kernel: i386 8254 system timestamp loses or gains whole ticks when IRQ 0 deliveries merge under CPU hold-off #61