[GH-ISSUE #68] fork: a child forked from a cthreads process intermittently dies with SIGSEGV (stress-thread fork) #63

New issue

Closed

opened 2026-11-03 22:03:34 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:34 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/68

Symptom

stress-thread fork failed once with a child process killed by SIGSEGV:

STRESS thread fork FAIL reason=child-status status=0xb worker=1 iter=14 seed=4

This was in the phase 4 --fs acceptance UFS run (2026-10-19, #60 kernel, ffs-perf 8235b390f), with every category running. Status 0xb is SIGSEGV.

What the test does

A worker starts several cthreads that loop locking a shared mutex_t and updating counters. The main thread fork()s while they run. The child does malloc(1 MB), fills and checks it, and calls _exit(0) (src/Commands/stress/thread/thread.c).

The child crashing, rather than hanging, rules out the lock-held-at-fork case that #49's analysis covered. fork.s already runs _cthread_fork_prepare/_cthread_fork_child and the dyld fork hooks.

Reproduce

Intermittent:

./thread/stress-thread -s 4 -n 4 fork, many iterations;
or run 1 with all categories under load.

Evidence: vm/work/ffsbench/runs/p4-accept/stress-0/thread.out. The test's core dump, if one was written, goes to /cores on the guest (the snapshot is gone, so rerun with cores enabled to get one).

Where to look

Child state after fork: the child's copy of the malloc zones or cthread state; a pointer valid only in a parent thread that doesn't exist in the child.
Kernel side: copy-on-write faults in the child for pages the parent's other threads were writing at fork time.
Debugging: reproduce with cores enabled and look at the faulting address and stack.

Related: #42 (fork exhaustion panic), #49 (fork "hang", a test bug). Found by the stress suite.

Originally created by @raynorpat on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/68 ## Symptom `stress-thread fork` failed once with a child process killed by SIGSEGV: ``` STRESS thread fork FAIL reason=child-status status=0xb worker=1 iter=14 seed=4 ``` This was in the phase 4 `--fs` acceptance UFS run (2026-10-19, #60 kernel, ffs-perf 8235b390f), with every category running. Status 0xb is SIGSEGV. ## What the test does A worker starts several cthreads that loop locking a shared `mutex_t` and updating counters. The main thread `fork()`s while they run. The child does `malloc(1 MB)`, fills and checks it, and calls `_exit(0)` (`src/Commands/stress/thread/thread.c`). The child crashing, rather than hanging, rules out the lock-held-at-fork case that #49's analysis covered. `fork.s` already runs `_cthread_fork_prepare`/`_cthread_fork_child` and the dyld fork hooks. ## Reproduce Intermittent: - `./thread/stress-thread -s 4 -n 4 fork`, many iterations; - or `run 1` with all categories under load. Evidence: `vm/work/ffsbench/runs/p4-accept/stress-0/thread.out`. The test's core dump, if one was written, goes to `/cores` on the guest (the snapshot is gone, so rerun with cores enabled to get one). ## Where to look - **Child state after fork:** the child's copy of the malloc zones or cthread state; a pointer valid only in a parent thread that doesn't exist in the child. - **Kernel side:** copy-on-write faults in the child for pages the parent's other threads were writing at fork time. - **Debugging:** reproduce with cores enabled and look at the faulting address and stack. Related: #42 (fork exhaustion panic), #49 (fork "hang", a test bug). Found by the stress suite.

gitea-mirror closed this issue

2026-11-03 22:03:35 -07:00

gitea-mirror commented

2026-11-03 22:03:36 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 20, 2026):

Fixed in Libc by 61ca43ca5 on ffs-perf (cherry-picked from issue-68 428749746).

It isn't #71. On a kernel with the FSAVE workaround (4fd6d07b6), stress-thread fork alone was clean: about 8,400 forks over 1 hour. Under run -a load the same failure came back (child-status status=0x8b) 3 times in about 140 rounds.

What the cores show. All 3 cores are the same:

The child crashes in _cthread_fork_child+0xa6 while it walks the cthread run queue, whose head is 0x80000000.
One of the parent's cprocs was still in cthread_body → condition_signal(&t->done) for a cthread t.
By then the main thread had already joined t, freed it, allocated it again and queued it.
The child frees t twice: once in _cproc_fork_child (free(p->incarnation), cprocs.c ~646) through the stale incarnation, and again from the run queue.

Root cause. In cthread_body (threads.subproj/cthreads.c ~204-212), the cproc sets T_RETURNED and drops t->lock before it signals t->done and before it clears its incarnation with cthread_assoc(self, NO_CTHREAD). The joiner can free t while the cproc still points at it.

Fix. The cproc clears its incarnation and signals the joiner while it still holds t->lock.

Verification. The test guests run DR2's System binary, not one built from src/Libc-1, so:

the Libc source builds with the change;
the same reordering was applied as a 69-byte patch to DR2's cthread_body, on snapshot guests only (vm/work/issue-68/patchsys.py);
with that patch: 250 rounds under run -a (about 20,000 forks), 0 fork failures, 0 cores;
check-thread passes.

A HEAD libsystem couldn't be built for the image (getsid/mlock/munlock syscalls, librpcsvc-obj missing), so the shipped binary itself hasn't been run.

The other failures under load (SIGSEGV in fdpass and stress-misc workers) were the fifo vnode reuse fixed by #67. Its kernel-fault-to-SIGSEGV path is #73.

@raynorpat commented on GitHub (Oct 20, 2026): Fixed in Libc by 61ca43ca5 on ffs-perf (cherry-picked from issue-68 428749746). **It isn't #71.** On a kernel with the FSAVE workaround (4fd6d07b6), `stress-thread fork` alone was clean: about 8,400 forks over 1 hour. Under `run -a` load the same failure came back (`child-status status=0x8b`) 3 times in about 140 rounds. **What the cores show.** All 3 cores are the same: - The child crashes in `_cthread_fork_child`+0xa6 while it walks the cthread run queue, whose head is 0x80000000. - One of the parent's cprocs was still in `cthread_body` → `condition_signal(&t->done)` for a cthread `t`. - By then the main thread had already joined `t`, freed it, allocated it again and queued it. - The child frees `t` twice: once in `_cproc_fork_child` (`free(p->incarnation)`, cprocs.c ~646) through the stale incarnation, and again from the run queue. **Root cause.** In `cthread_body` (`threads.subproj/cthreads.c` ~204-212), the cproc sets T_RETURNED and drops `t->lock` before it signals `t->done` and before it clears its incarnation with `cthread_assoc(self, NO_CTHREAD)`. The joiner can free `t` while the cproc still points at it. **Fix.** The cproc clears its incarnation and signals the joiner while it still holds `t->lock`. **Verification.** The test guests run DR2's System binary, not one built from `src/Libc-1`, so: - the Libc source builds with the change; - the same reordering was applied as a 69-byte patch to DR2's `cthread_body`, on snapshot guests only (`vm/work/issue-68/patchsys.py`); - with that patch: 250 rounds under `run -a` (about 20,000 forks), 0 fork failures, 0 cores; - `check-thread` passes. A HEAD libsystem couldn't be built for the image (getsid/mlock/munlock syscalls, librpcsvc-obj missing), so the shipped binary itself hasn't been run. The other failures under load (SIGSEGV in fdpass and stress-misc workers) were the fifo vnode reuse fixed by #67. Its kernel-fault-to-SIGSEGV path is #73.

gitea-mirror referenced this issue

2026-11-03 22:03:43 -07:00

[GH-ISSUE #79] ppc: PPCSerialPort finds no device on QEMU g3beige (OpenBIOS lists one SCC interrupt, not three) #70