github-starred/RhapsodiOS

Watch

Fork 0

mirror of https://github.com/RhapsodiOS/RhapsodiOS.git synced 2026-11-03 22:01:40 -07:00

[GH-ISSUE #40] kernel: i386 gettimeofday() steps backwards by a few microseconds #32

New issue

Closed

opened 2026-11-03 22:02:38 -07:00 by gitea-mirror · 3 comments

gitea-mirror commented

2026-11-03 22:02:38 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/40

Symptom

On i386 (QEMU TCG, -M pc -cpu pentium, 128 MB), user-space gettimeofday() sometimes returns a time a few microseconds earlier than a call made just before it in the same process.

The new stress suite caught it. stress-vm brk times each malloc/realloc/free with two gettimeofday() calls, and the latency is computed as an unsigned difference. A backward step wrapped it to maxlat=4294967ms. This happened on brk in both solo and concurrent runs on the v2 image.

Kernel: the v2 image's master + ffs-fixes kernel, kernel-155.0.0-7. It already contains the #26 fix, cd9fc5abd ("keep the i386 system clock monotonic across an unserviced PIT wrap").
Size of the step: a few microseconds, not the one 10 ms tick that #26 described. This looks like a different path, or the double-wrap case #26 left open.

Where to look

gettimeofday → microtime() (kern/ns_timer.c:144) → clock_get_counter(Calendar) (machdep/i386/machine_clock.c:281), which is system_time_stamp() + time_of_boot.

Candidates, unverified:

system_time_stamp() interpolation from the latched 8254 count. A count read just as the PIT reloads, or two wraps before IRQ0 is serviced (the limit #26 noted), can come out slightly lower.
clock_adjust_counter(Calendar, nsdelta) in hardclock (bsd/kern/kern_clock.c:257-267). With a negative tickdelta (an adjtime slew, e.g. from timed), each tick moves time_of_boot back by a few microseconds. That alone would produce microsecond-sized backward steps between two reads that straddle a tick.
clock_get_counter(Calendar) reads system_time_stamp() before taking splclock(), so a tick or adjust between the two can mix old and new values.

Reproduce

On the ffs-perf branch, with src/Commands/stress built on a v2 guest:

./vm/stress-vm -n 4 -i 20000 brk

Watch for maxlat= near 4294967 ms. A minimal user-space check is a loop of gettimeofday() that counts how often t2 < t1. I can add one to the suite.

Impact

Anything that measures intervals with gettimeofday() can see a negative interval, and unsigned arithmetic turns it into a huge one: timeouts, rate limiting, select/sleep emulation in user code, and benchmarks. The stress library now clamps negative intervals, but the clock itself should not go backwards except through an explicit settimeofday.

Related: #26 (one-tick backward step in IOGetTimestamp, fixed in cd9fc5abd), #28 (driver clamps).

Originally created by @raynorpat on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/40 ## Symptom On i386 (QEMU TCG, `-M pc -cpu pentium`, 128 MB), user-space `gettimeofday()` sometimes returns a time a few microseconds *earlier* than a call made just before it in the same process. The new stress suite caught it. `stress-vm brk` times each malloc/realloc/free with two `gettimeofday()` calls, and the latency is computed as an unsigned difference. A backward step wrapped it to `maxlat=4294967ms`. This happened on `brk` in both solo and concurrent runs on the v2 image. - **Kernel:** the v2 image's master + ffs-fixes kernel, kernel-155.0.0-7. It already contains the #26 fix, cd9fc5abd ("keep the i386 system clock monotonic across an unserviced PIT wrap"). - **Size of the step:** a few microseconds, not the one 10 ms tick that #26 described. This looks like a different path, or the double-wrap case #26 left open. ## Where to look `gettimeofday` → `microtime()` (`kern/ns_timer.c:144`) → `clock_get_counter(Calendar)` (`machdep/i386/machine_clock.c:281`), which is `system_time_stamp()` + `time_of_boot`. Candidates, unverified: - `system_time_stamp()` interpolation from the latched 8254 count. A count read just as the PIT reloads, or two wraps before IRQ0 is serviced (the limit #26 noted), can come out slightly lower. - `clock_adjust_counter(Calendar, nsdelta)` in `hardclock` (`bsd/kern/kern_clock.c:257-267`). With a negative `tickdelta` (an `adjtime` slew, e.g. from timed), each tick moves `time_of_boot` back by a few microseconds. That alone would produce microsecond-sized backward steps between two reads that straddle a tick. - `clock_get_counter(Calendar)` reads `system_time_stamp()` before taking `splclock()`, so a tick or adjust between the two can mix old and new values. ## Reproduce On the ffs-perf branch, with `src/Commands/stress` built on a v2 guest: ``` ./vm/stress-vm -n 4 -i 20000 brk ``` Watch for `maxlat=` near 4294967 ms. A minimal user-space check is a loop of `gettimeofday()` that counts how often t2 < t1. I can add one to the suite. ## Impact Anything that measures intervals with `gettimeofday()` can see a negative interval, and unsigned arithmetic turns it into a huge one: timeouts, rate limiting, `select`/`sleep` emulation in user code, and benchmarks. The stress library now clamps negative intervals, but the clock itself should not go backwards except through an explicit `settimeofday`. Related: #26 (one-tick backward step in `IOGetTimestamp`, fixed in cd9fc5abd), #28 (driver clamps).

gitea-mirror

2026-11-03 22:02:38 -07:00

closed this issue
added the
bug
label

gitea-mirror commented

2026-11-03 22:02:40 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 14, 2026):

Fix committed on master in e87c115e9 (not yet pushed or verified on a guest).

clock_get_counter() read system_time_stamp() before taking splclock(). A tick between the two could apply a negative adjtime step to time_of_boot after the stamp was read, pairing an old stamp with the new, lower base. The read and the time_of_boot add now happen in one splclock section.

Caveat: this race needs an adjtime slew in progress (e.g. timed). If the guest isn't slewing, the cause is elsewhere (PIT path). To verify: rebuild the kernel and run ./vm/stress-vm -n 4 -i 20000 brk; clockback (see #44) should read 0. Leaving open until that is confirmed.

@raynorpat commented on GitHub (Oct 14, 2026): Fix committed on master in e87c115e9 (not yet pushed or verified on a guest). `clock_get_counter()` read `system_time_stamp()` before taking `splclock()`. A tick between the two could apply a negative `adjtime` step to `time_of_boot` after the stamp was read, pairing an old stamp with the new, lower base. The read and the `time_of_boot` add now happen in one splclock section. Caveat: this race needs an adjtime slew in progress (e.g. timed). If the guest isn't slewing, the cause is elsewhere (PIT path). To verify: rebuild the kernel and run `./vm/stress-vm -n 4 -i 20000 brk`; `clockback` (see #44) should read 0. Leaving open until that is confirmed.

gitea-mirror commented

2026-11-03 22:02:41 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 14, 2026):

Test results for e87c115e9 (kernel = v2-kernel branch + this one-file change, kernel-155.0.0-7, QEMU TCG pentium).

Mechanism confirmed. A negative adjtime slew makes hardclock step time_of_boot by -tickadj (40 us) per tick; a tick landing between the stamp read and the time_of_boot add steps gettimeofday back by up to ~40 us. No slew and a positive slew gave back=0 on the stock kernel. A user loop (4 workers, 60 s, one process calling adjtime(-3000us) every ~2 ms):

stock kernel: 217, 250, 105 backward steps in 3 runs (worst 37-38 us), every worker affected
fixed kernel: 0, 0, 0 in 3 runs (15.0M, 15.4M, 8.4M calls)

Original stress symptom not fully explained. stress-vm -n 4 -i 20000 brk on the stock kernel showed clockback=2 in 1 of 4 runs, but nothing was running adjtime on the guest, so this race does not explain it. The fixed kernel had 8 clean runs, which is only ~10% likely by luck if the rate were still 1 in 4. Leaving this open until we know what, if anything, slews the clock on that guest or the brk repro is clean over more runs.

Separate finding: a kernel built from master e87c115e9 hangs at boot on the v2 guest (stops after 'minimum quantum is 10 ms'). v2-kernel has three commits master lacks (slave 8259 mask, fault-recovery vector clear, ffs_realloccg uncache); not bisected.

@raynorpat commented on GitHub (Oct 14, 2026): Test results for e87c115e9 (kernel = v2-kernel branch + this one-file change, kernel-155.0.0-7, QEMU TCG pentium). **Mechanism confirmed.** A negative `adjtime` slew makes hardclock step `time_of_boot` by -tickadj (40 us) per tick; a tick landing between the stamp read and the `time_of_boot` add steps gettimeofday back by up to ~40 us. No slew and a positive slew gave back=0 on the stock kernel. A user loop (4 workers, 60 s, one process calling `adjtime(-3000us)` every ~2 ms): - stock kernel: 217, 250, 105 backward steps in 3 runs (worst 37-38 us), every worker affected - fixed kernel: 0, 0, 0 in 3 runs (15.0M, 15.4M, 8.4M calls) **Original stress symptom not fully explained.** `stress-vm -n 4 -i 20000 brk` on the stock kernel showed `clockback=2` in 1 of 4 runs, but nothing was running adjtime on the guest, so this race does not explain it. The fixed kernel had 8 clean runs, which is only ~10% likely by luck if the rate were still 1 in 4. Leaving this open until we know what, if anything, slews the clock on that guest or the brk repro is clean over more runs. Separate finding: a kernel built from master e87c115e9 hangs at boot on the v2 guest (stops after 'minimum quantum is 10 ms'). v2-kernel has three commits master lacks (slave 8259 mask, fault-recovery vector clear, ffs_realloccg uncache); not bisected.

gitea-mirror commented

2026-11-03 22:02:41 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 14, 2026):

Fixed on master in e87c115e9 (atomic stamp + time_of_boot read) and cd33dd922 (system and calendar clock never read lower than their previous value; clock_set_counter takes splclock first and resets the clamp, so settimeofday can still step back). Not yet pushed.

Tested on a kernel-155.0.0-7 built from v2-kernel plus machine_clock.c (master itself does not boot on the v2 guest, see the note on this issue):

adjtime(-3000us) slew loop, 4 workers x 60 s: stock 217/250/105 backward steps; fixed 0/0/0 (18.0M/18.6M/10.7M calls).
stress-vm -n 4 -i 20000 brk, 20 runs: no clockback; a strict side loop comparing each gettimeofday with the previous made 330M calls with 0 backward steps.
settimeofday steps back and forward correctly; guest/host time ratio 1.002; sleep 10 takes ~10 s.

The rare stock stress-vm clockback (about 1 run in 12, no adjtime in use) is probably a second PIT reload before IRQ0 is serviced; the new clamp holds the clock still in that case. It does not recover lost ticks, and I did not trace the cause. Reopen if it shows up again.

@raynorpat commented on GitHub (Oct 14, 2026): Fixed on master in e87c115e9 (atomic stamp + time_of_boot read) and cd33dd922 (system and calendar clock never read lower than their previous value; clock_set_counter takes splclock first and resets the clamp, so settimeofday can still step back). Not yet pushed. Tested on a kernel-155.0.0-7 built from v2-kernel plus machine_clock.c (master itself does not boot on the v2 guest, see the note on this issue): - adjtime(-3000us) slew loop, 4 workers x 60 s: stock 217/250/105 backward steps; fixed 0/0/0 (18.0M/18.6M/10.7M calls). - stress-vm -n 4 -i 20000 brk, 20 runs: no clockback; a strict side loop comparing each gettimeofday with the previous made 330M calls with 0 backward steps. - settimeofday steps back and forward correctly; guest/host time ratio 1.002; sleep 10 takes ~10 s. The rare stock stress-vm clockback (about 1 run in 12, no adjtime in use) is probably a second PIT reload before IRQ0 is serviced; the new clamp holds the clock still in that case. It does not recover lost ticks, and I did not trace the cause. Reopen if it shows up again.

gitea-mirror referenced this issue

2026-11-03 22:04:34 -07:00

[PR #32] [MERGED] kernel: finish Xoodyak entropy integration #88