[GH-ISSUE #66] nfs: loopback client loses O_APPEND data, returns ESTALE/EEXIST after local renames, and gives no SIGBUS past a truncated mapping #58

New issue

Closed

opened 2026-11-03 22:03:31 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:31 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/66

Symptom

A loopback NFS mount fails several stress tests with only one client and only local processes:

the server is nfsd/mountd on the same guest;
the export is /mnt1/nfsexp on UFS, published through NetInfo /exports;
it is mounted with mount_nfs localhost:/mnt1/nfsexp /mnt/nfs.

Results in two runs (p4-nfs2 on the issue58 kernel, p4-accept on the #60 kernel):

fs append, wrong size: FAIL what=size off=0xc31f8 / 0xc3200. Four processes O_APPEND-write to one file on the same client, and the final size is short.
fs tree / rename / link:
- unlink: ESTALE, open: ESTALE;
- open: EEXIST on a name the test had just removed;
- rename ... short-read.
mmap trunc, no SIGBUS: FAIL reason=no-sigbus. After ftruncate shrinks a mapped NFS file, touching the mapping past the new EOF does not fault.
mmap regrow, stale data: what=zero off=0x5000 (issue58 kernel), the #59 symptom on NFS. Recheck it on a kernel with the #59 fix.

21 pass, 9 fail on the #60 kernel.

Reproduce

python vm/ffsbench.py stress --kernel <kernel> --label x --fs nfs --seed 4

This uses branch stress-phase4's vm/stress_fs.sh NFS target, which does the NetInfo export, starts mountd and nfsd -tu -n 4, and mounts localhost. Evidence: vm/work/ffsbench/runs/p4-nfs2/stress-0/ and p4-accept/stress-0/ (nfs.*.out).

Where to look

append: the NFS client's O_APPEND path (nfs_write). The offset is taken from cached attributes (np->n_size) without serialising against other local writers, or without refreshing the size.
ESTALE/EEXIST: the client name cache and attribute cache after local rename/unlink (nfs_lookup, cache_purge on the client side).
trunc/regrow:
- nfs_setattr with a smaller size must call vnode_pager_setsize/the mapfs truncate the way ffs_truncate does;
- #59's vno_flush_eof treatment may be needed for NFS's mapped vnodes.

Related: #41, #59. Found by stress suite phase 4's --fs nfs target.

Originally created by @raynorpat on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/66 ## Symptom A loopback NFS mount fails several stress tests with only one client and only local processes: - the server is `nfsd`/`mountd` on the same guest; - the export is `/mnt1/nfsexp` on UFS, published through NetInfo `/exports`; - it is mounted with `mount_nfs localhost:/mnt1/nfsexp /mnt/nfs`. Results in two runs (`p4-nfs2` on the issue58 kernel, `p4-accept` on the #60 kernel): - **`fs append`, wrong size:** `FAIL what=size off=0xc31f8` / `0xc3200`. Four processes `O_APPEND`-write to one file on the same client, and the final size is short. - **`fs tree` / `rename` / `link`:** - `unlink: ESTALE`, `open: ESTALE`; - `open: EEXIST` on a name the test had just removed; - `rename ... short-read`. - **`mmap trunc`, no SIGBUS:** `FAIL reason=no-sigbus`. After `ftruncate` shrinks a mapped NFS file, touching the mapping past the new EOF does not fault. - **`mmap regrow`, stale data:** `what=zero off=0x5000` (issue58 kernel), the #59 symptom on NFS. Recheck it on a kernel with the #59 fix. 21 pass, 9 fail on the #60 kernel. ## Reproduce ``` python vm/ffsbench.py stress --kernel <kernel> --label x --fs nfs --seed 4 ``` This uses branch `stress-phase4`'s `vm/stress_fs.sh` NFS target, which does the NetInfo export, starts `mountd` and `nfsd -tu -n 4`, and mounts `localhost`. Evidence: `vm/work/ffsbench/runs/p4-nfs2/stress-0/` and `p4-accept/stress-0/` (`nfs.*.out`). ## Where to look - **append:** the NFS client's `O_APPEND` path (`nfs_write`). The offset is taken from cached attributes (`np->n_size`) without serialising against other local writers, or without refreshing the size. - **ESTALE/EEXIST:** the client name cache and attribute cache after local `rename`/`unlink` (`nfs_lookup`, `cache_purge` on the client side). - **trunc/regrow:** - `nfs_setattr` with a smaller size must call `vnode_pager_setsize`/the mapfs truncate the way `ffs_truncate` does; - #59's `vno_flush_eof` treatment may be needed for NFS's mapped vnodes. Related: #41, #59. Found by stress suite phase 4's `--fs nfs` target.

gitea-mirror referenced this issue

2026-11-03 22:03:31 -07:00

[GH-ISSUE #65] msdosfs: concurrent writes corrupt FAT16 data and the FAT (EINVAL, wrong blocks, lost clusters) and panic 'extendfile: pcbmap' #59

gitea-mirror closed this issue

2026-11-03 22:03:32 -07:00

gitea-mirror commented

2026-11-03 22:03:33 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 20, 2026):

Fixed on ffs-perf (merge b37bece0d of branch issue-66, five commits).

Not #71: every symptom reproduced on a kernel with the FSAVE workaround (ffs-perf 4fd6d07b6).

Fixes:

161d8004f, O_APPEND size loss (client bug). NFS has no vnode lock (nfs_lock is vop_nolock), so two O_APPEND writers could read the same size and write at the same end. nfs_write (nfs_bio.c) now runs appenders one at a time, in arrival order (a ticket queue; its counters reuse a union member regular files don't use, so nfsnode stays 256 bytes).
7aae56e80, no SIGBUS past a truncated mapping, and regrow stale data (client bug). nfs_setattr (nfs_vnops.c ~727) called mapfs_trunc only when MapFS had a window open, so a user mapping kept its pages past the new end and #59's vno_flush_eof never ran for NFS. It now truncates the way ffs_truncate does.
be0ad837d, EEXIST after a retried exclusive create (server side). nfsrv_create stores the client's verifier in atime, but ufs_setattr (ufs_vnops.c ~484) dropped the nanoseconds. A retry that missed the 64-entry reply cache then got EEXIST; 6-13 retries were seen per run.
84e363177, EIO on create or mkdir in a removed directory (server quirk). UFS's ENOENT was mapped to IO by nfsrv_errmap (nfs_subs.c ~2219). It now answers STALE.
c0bc4e524, ESTALE is inherent, not a bug. Another process removes the object between a lookup and the call that uses its file handle; without vnode locks the client can't prevent that. The stress fs tests now end UNSUP with reason="ESTALE: object removed by another worker" (README updated).

Verified on a kernel with these fixes (cksum 3567260633):

--fs nfs seed 4: 29 PASS, 1 UNSUP, 0 FAIL.
Seed 7: 28 PASS, 2 UNSUP, 0 FAIL.
Earlier builds with the same fixes: 0 FAIL at seeds 4 (twice), 7 and 11. Before: 9 FAIL.
UFS check-fs and check-mmap pass.

Left open:

One run, on an intermediate build, ended with umount: /mnt/nfs: Device busy even with -f. It did not recur in five later runs; cause unknown.
NFS append now runs about 60-100 ops/s.
tree usually ends UNSUP on NFS, so it says little about NFS.

@raynorpat commented on GitHub (Oct 20, 2026): Fixed on ffs-perf (merge b37bece0d of branch issue-66, five commits). **Not #71:** every symptom reproduced on a kernel with the FSAVE workaround (ffs-perf 4fd6d07b6). **Fixes:** 1. **161d8004f, O_APPEND size loss (client bug).** NFS has no vnode lock (nfs_lock is vop_nolock), so two O_APPEND writers could read the same size and write at the same end. `nfs_write` (`nfs_bio.c`) now runs appenders one at a time, in arrival order (a ticket queue; its counters reuse a union member regular files don't use, so `nfsnode` stays 256 bytes). 2. **7aae56e80, no SIGBUS past a truncated mapping, and regrow stale data (client bug).** `nfs_setattr` (`nfs_vnops.c` ~727) called `mapfs_trunc` only when MapFS had a window open, so a user mapping kept its pages past the new end and #59's `vno_flush_eof` never ran for NFS. It now truncates the way `ffs_truncate` does. 3. **be0ad837d, EEXIST after a retried exclusive create (server side).** `nfsrv_create` stores the client's verifier in atime, but `ufs_setattr` (`ufs_vnops.c` ~484) dropped the nanoseconds. A retry that missed the 64-entry reply cache then got EEXIST; 6-13 retries were seen per run. 4. **84e363177, EIO on create or mkdir in a removed directory (server quirk).** UFS's ENOENT was mapped to IO by `nfsrv_errmap` (`nfs_subs.c` ~2219). It now answers STALE. 5. **c0bc4e524, ESTALE is inherent, not a bug.** Another process removes the object between a lookup and the call that uses its file handle; without vnode locks the client can't prevent that. The stress fs tests now end UNSUP with `reason="ESTALE: object removed by another worker"` (README updated). **Verified** on a kernel with these fixes (cksum 3567260633): - `--fs nfs` seed 4: 29 PASS, 1 UNSUP, 0 FAIL. - Seed 7: 28 PASS, 2 UNSUP, 0 FAIL. - Earlier builds with the same fixes: 0 FAIL at seeds 4 (twice), 7 and 11. Before: 9 FAIL. - UFS `check-fs` and `check-mmap` pass. **Left open:** - One run, on an intermediate build, ended with `umount: /mnt/nfs: Device busy` even with `-f`. It did not recur in five later runs; cause unknown. - NFS append now runs about 60-100 ops/s. - `tree` usually ends UNSUP on NFS, so it says little about NFS.