[GH-ISSUE #65] msdosfs: concurrent writes corrupt FAT16 data and the FAT (EINVAL, wrong blocks, lost clusters) and panic 'extendfile: pcbmap' #59

New issue

Closed

opened 2026-11-03 22:03:31 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:31 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/65

Symptom

Under the stress suite's concurrent fs load, a FAT16 volume on the i386 kernel corrupts data and its FAT. Under fs and mmap load together it panics. Seen across five harness runs between 2026-10-16 and 2026-10-19, on the issue58 kernel (ffs-perf with #52-#56 and #58) and on the #60 kernel (ffs-perf 8235b390f: plus #59 and #60):

Wrong data: STRESS fs fsync FAIL what=block off=0x3800 want=94b44c7b... got=604ab9f3..., a block read back with someone else's data.
EINVAL from the file system:
- STRESS fs rw FAIL reason="read: EINVAL" and reason="write: EINVAL", several workers and seeds;
- STRESS mmap trunc FAIL reason="ftruncate: EINVAL".
FAT damage after unmount:
- fsck_msdos -n reports lost cluster chains;
- in another run, ... starts with free cluster, has entries after end of directory, and a garbage name.
Panic: panic: extendfile: pcbmap, twice, both with fs and mmap in the same run. Running fs alone or mmap alone did not panic.

Reproduce

The phase 4 harness (branch stress-phase4):

python vm/ffsbench.py stress --kernel <kernel> --label x --fs msdos --seed 4

This formats partition c of a 3-partition targets disk (512-byte NeXT label, 128 MB) with the guest-built newfs_msdos (FAT16), mounts it at /mnt/msdos, runs run -s 4 -d /mnt/msdos 1 fs mmap and then fscks it. Use --cats fs for the corruption without the panic.

Evidence:

vm/work/ffsbench/runs/p4-accept/stress-0/: msdos.*.out, msdos.summary, the targets disk image, stressfs.out
p4-msfs/ (fs only), p4-msmm/ (mmap only)
p4-fs3/ and p4-fs4/ (the panics; serial logs)

Where to look

The panic: extendfile in bsd/msdosfs/msdosfs_fat.c, where pcbmap fails while the file is being extended.
Shared state: the corruption suggests concurrent cluster allocation or FAT updates aren't serialised. Several workers extend different files, and fsync then reads back a block that now belongs to another file.
Recent changes to compare:
- the #52-#55 msdosfs rework (260033acb: rename locking, msdosfs_lookup, pageout and size);
- #58's block-size scaling (883ed93c2).
Check extendfile/clusteralloc locking against concurrent deextend and detrunc.

Related: #54, #55, #58. Found by stress suite phase 4's --fs targets.

Originally created by @raynorpat on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/65 ## Symptom Under the stress suite's concurrent `fs` load, a FAT16 volume on the i386 kernel corrupts data and its FAT. Under `fs` and `mmap` load together it panics. Seen across five harness runs between 2026-10-16 and 2026-10-19, on the issue58 kernel (ffs-perf with #52-#56 and #58) and on the #60 kernel (ffs-perf 8235b390f: plus #59 and #60): - **Wrong data:** `STRESS fs fsync FAIL what=block off=0x3800 want=94b44c7b... got=604ab9f3...`, a block read back with someone else's data. - **EINVAL from the file system:** - `STRESS fs rw FAIL reason="read: EINVAL"` and `reason="write: EINVAL"`, several workers and seeds; - `STRESS mmap trunc FAIL reason="ftruncate: EINVAL"`. - **FAT damage after unmount:** - `fsck_msdos -n` reports lost cluster chains; - in another run, `... starts with free cluster`, `has entries after end of directory`, and a garbage name. - **Panic:** `panic: extendfile: pcbmap`, twice, both with `fs` and `mmap` in the same run. Running `fs` alone or `mmap` alone did not panic. ## Reproduce The phase 4 harness (branch `stress-phase4`): ``` python vm/ffsbench.py stress --kernel <kernel> --label x --fs msdos --seed 4 ``` This formats partition c of a 3-partition targets disk (512-byte NeXT label, 128 MB) with the guest-built `newfs_msdos` (FAT16), mounts it at `/mnt/msdos`, runs `run -s 4 -d /mnt/msdos 1 fs mmap` and then fscks it. Use `--cats fs` for the corruption without the panic. Evidence: - `vm/work/ffsbench/runs/p4-accept/stress-0/`: `msdos.*.out`, `msdos.summary`, the targets disk image, `stressfs.out` - `p4-msfs/` (fs only), `p4-msmm/` (mmap only) - `p4-fs3/` and `p4-fs4/` (the panics; serial logs) ## Where to look - **The panic:** `extendfile` in `bsd/msdosfs/msdosfs_fat.c`, where `pcbmap` fails while the file is being extended. - **Shared state:** the corruption suggests concurrent cluster allocation or FAT updates aren't serialised. Several workers extend different files, and `fsync` then reads back a block that now belongs to another file. - **Recent changes to compare:** - the #52-#55 msdosfs rework (260033acb: rename locking, `msdosfs_lookup`, pageout and size); - #58's block-size scaling (883ed93c2). Check `extendfile`/`clusteralloc` locking against concurrent `deextend` and `detrunc`. Related: #54, #55, #58. Found by stress suite phase 4's `--fs` targets.

gitea-mirror closed this issue

2026-11-03 22:03:32 -07:00

gitea-mirror referenced this issue

2026-11-03 22:03:33 -07:00

[GH-ISSUE #66] nfs: loopback client loses O_APPEND data, returns ESTALE/EEXIST after local renames, and gives no SIGBUS past a truncated mapping #58

gitea-mirror commented

2026-11-03 22:03:33 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 20, 2026):

Fixed by 56d820206 (kernel: recheck the buffer hash after getnewbuf sleeps in getblk, so two threads cannot make duplicate buffers for one block), merged into ffs-perf in 306504515.

Not #71: it reproduced on the first run on a kernel with the FSAVE workaround (ffs-perf 4fd6d07b6).

Cause: the buffer cache, not msdosfs.

On a cache miss, getblk() (bsd/vfs/vfs_bio.c ~698) called getnewbuf(), which can sleep writing out a delayed buffer, and never re-checked the hash afterwards.
It then ran allocbuf(), which can also sleep, before the new buffer was findable (block number and vnode not yet set).
In either window a second thread could create another buffer for the same block. With two copies of one FAT block, one thread's update is lost.

Evidence:

A debug kernel saw FAT[6144] (the first entry of FAT block 25) read back free right after fatchain had written a contiguous run through it.
pcbmap then followed the broken chain to cluster 0, which is inside the root directory: hence the garbage names and "entries after end of directory".
The chain then reached 0xFFF0, past the end of the partition: hence the EINVAL from the disk layer.
A recheck counter (GETBLK raced) caught the race on FAT blocks 1, 9, 57, 177, 185 and 225.
The extendfile: pcbmap panic and the lost chains are the same lost update.

Fix:

Re-check the hash after getnewbuf(); if the block appeared meanwhile, drop the new buffer and start over.
Make the busy buffer findable (block number, vnode, hash) before allocbuf(), so other threads wait on it instead of duplicating it.

The race is in code every buffer-cache filesystem shares, so UFS, ext2 and HFS benefit too.

Verified on the fixed kernel (cksum 1947222504):

msdos fs mmap at seeds 4, 9 and 7: PASS=28, FAIL=0, UNSUP=2 each, fsck_msdos clean, no panic.
ext2 PASS=30 and hfs PASS=26 UNSUP=4, both fsck clean.
UFS fs PASS=12, mmap PASS=18.

Note: msdos fs rw passes but takes about 1000 s on this host, over the harness's default 600 s per-test watchdog, so default --fs msdos runs still report an rw timeout. That's a speed matter, not this bug.

@raynorpat commented on GitHub (Oct 20, 2026): Fixed by 56d820206 (kernel: recheck the buffer hash after getnewbuf sleeps in getblk, so two threads cannot make duplicate buffers for one block), merged into ffs-perf in 306504515. **Not #71:** it reproduced on the first run on a kernel with the FSAVE workaround (ffs-perf 4fd6d07b6). **Cause: the buffer cache, not msdosfs.** - On a cache miss, `getblk()` (`bsd/vfs/vfs_bio.c` ~698) called `getnewbuf()`, which can sleep writing out a delayed buffer, and never re-checked the hash afterwards. - It then ran `allocbuf()`, which can also sleep, before the new buffer was findable (block number and vnode not yet set). - In either window a second thread could create another buffer for the same block. With two copies of one FAT block, one thread's update is lost. **Evidence:** - A debug kernel saw `FAT[6144]` (the first entry of FAT block 25) read back free right after `fatchain` had written a contiguous run through it. - `pcbmap` then followed the broken chain to cluster 0, which is inside the root directory: hence the garbage names and "entries after end of directory". - The chain then reached 0xFFF0, past the end of the partition: hence the EINVAL from the disk layer. - A recheck counter (`GETBLK raced`) caught the race on FAT blocks 1, 9, 57, 177, 185 and 225. - The `extendfile: pcbmap` panic and the lost chains are the same lost update. **Fix:** - Re-check the hash after `getnewbuf()`; if the block appeared meanwhile, drop the new buffer and start over. - Make the busy buffer findable (block number, vnode, hash) before `allocbuf()`, so other threads wait on it instead of duplicating it. The race is in code every buffer-cache filesystem shares, so UFS, ext2 and HFS benefit too. **Verified** on the fixed kernel (cksum 1947222504): - msdos `fs mmap` at seeds 4, 9 and 7: PASS=28, FAIL=0, UNSUP=2 each, fsck_msdos clean, no panic. - ext2 PASS=30 and hfs PASS=26 UNSUP=4, both fsck clean. - UFS fs PASS=12, mmap PASS=18. **Note:** msdos `fs rw` passes but takes about 1000 s on this host, over the harness's default 600 s per-test watchdog, so default `--fs msdos` runs still report an `rw` timeout. That's a speed matter, not this bug.