[GH-ISSUE #86] kernel: mmap and map_fd of an active swap file now fail instead of mapping a null pager #80

New issue

Closed

opened 2026-11-03 22:04:16 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:04:16 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/86

Side effect of #84 (eba20b6c5).

vnode_pager_setup (vm/vnode_pager.c ~657-665) returns vm_pager_null for a vnode in use as a pager file (swap file) on the pager_files list. That was always true. Before #84, the callers didn't check:

mmap (bsd/kern/kern_mman.c ~342) and map_fd (~791) went on and mapped the file with a null pager.

Since #84 they treat a null pager as "reclaimed":

mmap returns EBADF;
map_fd returns KERN_INVALID_ARGUMENT.

Two things to decide:

The intended behaviour. Mapping an active swap file with a null pager was almost certainly broken: the mapping would get anonymous zero-fill pages, not the file's contents, and could hide data. Failing looks right, but EBADF is the reclaimed-vnode errno. A distinct errno (EPERM, ETXTBSY or EACCES) might be clearer, and the two cases (swap file vs reclaimed) should be told apart in vnode_pager_setup's callers.
A pre-existing bug on the same path: the pager-file early return in vnode_pager_setup skips unix_release(), unlike the other returns. It leaves the unix master lock held.

Check who maps swap files in practice (dynamic_pager / macx_swapon, vm_stat-style tools) so nothing that worked before breaks.

Originally created by @raynorpat on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/86 Side effect of #84 (eba20b6c5). `vnode_pager_setup` (`vm/vnode_pager.c` ~657-665) returns `vm_pager_null` for a vnode in use as a pager file (swap file) on the `pager_files` list. That was always true. Before #84, the callers didn't check: - `mmap` (`bsd/kern/kern_mman.c` ~342) and `map_fd` (~791) went on and mapped the file with a null pager. Since #84 they treat a null pager as "reclaimed": - `mmap` returns EBADF; - `map_fd` returns KERN_INVALID_ARGUMENT. Two things to decide: 1. **The intended behaviour.** Mapping an active swap file with a null pager was almost certainly broken: the mapping would get anonymous zero-fill pages, not the file's contents, and could hide data. Failing looks right, but EBADF is the reclaimed-vnode errno. A distinct errno (EPERM, ETXTBSY or EACCES) might be clearer, and the two cases (swap file vs reclaimed) should be told apart in `vnode_pager_setup`'s callers. 2. **A pre-existing bug on the same path:** the pager-file early return in `vnode_pager_setup` skips `unix_release()`, unlike the other returns. It leaves the unix master lock held. Check who maps swap files in practice (dynamic_pager / macx_swapon, `vm_stat`-style tools) so nothing that worked before breaks.

gitea-mirror closed this issue

2026-11-03 22:04:17 -07:00

gitea-mirror commented

2026-11-03 22:04:18 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 31, 2026):

Fixed by 1d3cb1f80, merged into ffs-perf in 23c1dd5f6.

Who maps swap files.

At boot, /etc/startup/0500_VirtualMemory runs mach_swapon -av from /etc/swaptab (/private/var/vm/swapfile).
Nothing in the tree mmaps or map_fd's a swap file, except cp: file_cmds/cp/utils.c :152 mmaps any file of 8 MB or less.
- Before #84, cp of a small swap file silently copied zero-fill pages.
- After #84 it failed with "Bad file descriptor".
- Now it fails with "Text file busy".
- A swap file over 8 MB still copies with read().

Telling the cases apart. vnode_pager_setup returns NULL in both cases, so callers check v_vm_info. The function comment now says so.

Active swap file (v_vm_info set): mmap → ETXTBSY, map_fd → KERN_PROTECTION_FAILURE. ETXTBSY because the kernel is using the file as backing store, which is what vn_writechk already uses ETXTBSY for (vfs_vnops.c :226). The condition is temporary and applies to root too. EACCES would wrongly point at permissions.
Reclaimed vnode (v_vm_info NULL): unchanged from #84, EBADF / KERN_INVALID_ARGUMENT.

exec of a swap file: EACCES at mode 1600, otherwise ENOEXEC (not Mach-O).

unix_release: the pager-file early return in vnode_pager_setup (:663) now releases the unix master. The other returns of vnode_pager_setup and vnode_pager_create were already balanced. On these NCPUS 1 builds unix_master is a no-op, so the missing call only mattered for a multi-CPU kernel.

Verified (kernel cksum 151843857):

Base: mmap EBADF, map_fd 4.
Fix: mmap ETXTBSY, map_fd 2, on the boot swap file and on a second one added with mach_swapon. Ordinary files still map.
Swap at boot and under stress-vm pressure (swap growing to about 420 MB) works.
The #81/#84 recipes and the check targets pass, as in #85.

Left open: map_vnode (mapfs.c :499) still carries on with a NULL pager for an active swap file. This is old behaviour, and read() of the swap file works.

@raynorpat commented on GitHub (Oct 31, 2026): Fixed by 1d3cb1f80, merged into ffs-perf in 23c1dd5f6. **Who maps swap files.** - At boot, `/etc/startup/0500_VirtualMemory` runs `mach_swapon -av` from `/etc/swaptab` (`/private/var/vm/swapfile`). - Nothing in the tree mmaps or map_fd's a swap file, except `cp`: `file_cmds/cp/utils.c` :152 mmaps any file of 8 MB or less. - Before #84, `cp` of a small swap file silently copied zero-fill pages. - After #84 it failed with "Bad file descriptor". - Now it fails with "Text file busy". - A swap file over 8 MB still copies with `read()`. **Telling the cases apart.** `vnode_pager_setup` returns NULL in both cases, so callers check `v_vm_info`. The function comment now says so. - **Active swap file** (`v_vm_info` set): mmap → **ETXTBSY**, map_fd → **KERN_PROTECTION_FAILURE**. ETXTBSY because the kernel is using the file as backing store, which is what `vn_writechk` already uses ETXTBSY for (`vfs_vnops.c` :226). The condition is temporary and applies to root too. EACCES would wrongly point at permissions. - **Reclaimed vnode** (`v_vm_info` NULL): unchanged from #84, EBADF / KERN_INVALID_ARGUMENT. **exec of a swap file:** EACCES at mode 1600, otherwise ENOEXEC (not Mach-O). **`unix_release`:** the pager-file early return in `vnode_pager_setup` (:663) now releases the unix master. The other returns of `vnode_pager_setup` and `vnode_pager_create` were already balanced. On these NCPUS 1 builds `unix_master` is a no-op, so the missing call only mattered for a multi-CPU kernel. **Verified** (kernel cksum 151843857): - **Base:** mmap EBADF, map_fd 4. - **Fix:** mmap ETXTBSY, map_fd 2, on the boot swap file and on a second one added with `mach_swapon`. Ordinary files still map. - **Swap at boot and under `stress-vm pressure`** (swap growing to about 420 MB) works. - The #81/#84 recipes and the check targets pass, as in #85. **Left open:** `map_vnode` (`mapfs.c` :499) still carries on with a NULL pager for an active swap file. This is old behaviour, and `read()` of the swap file works.

gitea-mirror referenced this issue

2026-11-04 22:03:16 -07:00

[GH-ISSUE #80] I knew someone else would be doing this too lol. #73