[GH-ISSUE #69] kernel: i386 8254 system timestamp loses or gains whole ticks when IRQ 0 deliveries merge under CPU hold-off #61

New issue

Closed

opened 2026-11-03 22:03:33 -07:00 by gitea-mirror · 1 comment

gitea-mirror commented

2026-11-03 22:03:33 -07:00

Owner

Originally created by @raynorpat on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/69

Symptom

On i386, the kernel's 8254-based clock (IOGetTimestamp → clock_get_counter/system_time_stamp, kernel-7/machdep/i386/machine_clock.c ~450-490) loses or gains whole ticks when the CPU is held off. Under QEMU TCG on a busy host that happens when the host deschedules the vCPU.

Found while fixing #64:

Lost tick: if the CPU is held off across two counter-0 reloads, the edges merge into one IRQ 0 and a whole tick (10 ms) is lost. The timestamp only notices a reload it saw happen (current > last, ~:464-473).
Frozen clock: the "hold still" clamp (~:482-486) then freezes the timestamp for up to a tick.
Jump: a tick interrupt arriving after a read makes the next read jump.
Measured: a probe saw a 10.03 ms timestamp span 19.9 ms of LAPIC time.

pexpert's LAPIC calibration no longer depends on this (fixed in #64), and with the LAPIC clock active the system clock no longer uses it. But systems that stay on the 8254 clock, and every user of IOGetTimestamp/the system timestamp, still see time stand still or jump under host scheduling delays.

Expected

The 8254-based timestamp stays monotonic and doesn't lose time when IRQ 0 deliveries merge or are delayed. Either detect missed reloads (for example by comparing against another free-running reference) and account for them, or derive time from a source that can't lose ticks.

Related: #64, #46, #40 (gettimeofday stepping back).

Originally created by @raynorpat on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/RhapsodiOS/RhapsodiOS/issues/69 ## Symptom On i386, the kernel's 8254-based clock (`IOGetTimestamp` → `clock_get_counter`/`system_time_stamp`, `kernel-7/machdep/i386/machine_clock.c` ~450-490) loses or gains whole ticks when the CPU is held off. Under QEMU TCG on a busy host that happens when the host deschedules the vCPU. Found while fixing #64: - **Lost tick:** if the CPU is held off across two counter-0 reloads, the edges merge into one IRQ 0 and a whole tick (10 ms) is lost. The timestamp only notices a reload it saw happen (`current > last`, ~:464-473). - **Frozen clock:** the "hold still" clamp (~:482-486) then freezes the timestamp for up to a tick. - **Jump:** a tick interrupt arriving after a read makes the next read jump. - **Measured:** a probe saw a 10.03 ms timestamp span 19.9 ms of LAPIC time. pexpert's LAPIC calibration no longer depends on this (fixed in #64), and with the LAPIC clock active the system clock no longer uses it. But systems that stay on the 8254 clock, and every user of `IOGetTimestamp`/the system timestamp, still see time stand still or jump under host scheduling delays. ## Expected The 8254-based timestamp stays monotonic and doesn't lose time when IRQ 0 deliveries merge or are delayed. Either detect missed reloads (for example by comparing against another free-running reference) and account for them, or derive time from a source that can't lose ticks. Related: #64, #46, #40 (gettimeofday stepping back).

gitea-mirror closed this issue

2026-11-03 22:03:34 -07:00

gitea-mirror commented

2026-11-03 22:03:35 -07:00

Author

Owner

@raynorpat commented on GitHub (Oct 20, 2026):

Fixed by a7fbf3287 (kernel: count i386 tick reloads with the TSC so ticks merged under interrupt hold-off are not lost), merged into ffs-perf in d0eb6a33b.

Mechanism.

system_timer_dispatch added exactly one tick per interrupt. A CPU held off across two or more reloads gets one merged interrupt, and the other reloads are lost for good. This affects both the 8254 (IRQ 0) and the LAPIC periodic timer.
system_time_stamp only noticed a single wrap, and the monotonic clamp then held the stamp still.
The same merge happens through the soft-IPL defer table when IRQ 0 arrives at or above splclock.

Fix (machdep/i386/machine_clock.c only).

Calibration: the TSC is calibrated at machine_clock_init against 8254 counter 2, using bracketed reads, refusing a sample if counter 2 ran out, with up to 5 tries. CPUID's TSC bit is checked first. With no TSC, or no clean sample, the old code path runs unchanged.
Counting reloads: the tick handler and the timestamp count the counter reloads since the last one counted, from the TSC time of the latest reload, read between two TSC reads.
No drift: the reference resets every tick, so calibration error can't build up. A read that was itself held off falls back to the old behaviour.
Unchanged: the monotonic and #40 clamps. The reference is cleared on the LAPIC switch.

Hold-off probe (interrupts off for N ms, timed against counter 2; 6 concurrent boots × 3 runs per row, both clock paths):

hold	stock 8254	stock LAPIC	fixed, both paths
15 ms	10/18 lost 10 ms	9/18 lost 10–20 ms	lost 5–49 µs
25 ms	18/18 lost 10–20 ms	17/17 lost 10–20 ms	lost 5–49 µs
35 ms	18/18 lost 20–30 ms	17/17 lost 20–30 ms	lost 5–49 µs

Drift, 6 concurrent guests over 300 s with a busy vCPU each:

8254 path: stock −0.045 to −1.584 s, fixed −0.005 s on all six.
LAPIC path: stock −0.168 to −3.287 s, fixed −0.019 to −0.030 s.

misc timer, vm clock (no backward steps), make check-misc and check-vm pass on both paths.

Left open:

The fix assumes a constant-rate TSC. A CPU whose TSC changes with clock speed would miscount reloads.
hardclock's scheduler and statistics accounting still sees one call per interrupt; only the clock is corrected.

@raynorpat commented on GitHub (Oct 20, 2026): Fixed by a7fbf3287 (kernel: count i386 tick reloads with the TSC so ticks merged under interrupt hold-off are not lost), merged into ffs-perf in d0eb6a33b. **Mechanism.** - `system_timer_dispatch` added exactly one tick per interrupt. A CPU held off across two or more reloads gets one merged interrupt, and the other reloads are lost for good. This affects both the 8254 (IRQ 0) and the LAPIC periodic timer. - `system_time_stamp` only noticed a single wrap, and the monotonic clamp then held the stamp still. - The same merge happens through the soft-IPL defer table when IRQ 0 arrives at or above splclock. **Fix** (`machdep/i386/machine_clock.c` only). - **Calibration:** the TSC is calibrated at `machine_clock_init` against 8254 counter 2, using bracketed reads, refusing a sample if counter 2 ran out, with up to 5 tries. CPUID's TSC bit is checked first. With no TSC, or no clean sample, the old code path runs unchanged. - **Counting reloads:** the tick handler and the timestamp count the counter reloads since the last one counted, from the TSC time of the latest reload, read between two TSC reads. - **No drift:** the reference resets every tick, so calibration error can't build up. A read that was itself held off falls back to the old behaviour. - **Unchanged:** the monotonic and #40 clamps. The reference is cleared on the LAPIC switch. **Hold-off probe** (interrupts off for N ms, timed against counter 2; 6 concurrent boots × 3 runs per row, both clock paths): | hold | stock 8254 | stock LAPIC | fixed, both paths | |---|---|---|---| | 15 ms | 10/18 lost 10 ms | 9/18 lost 10–20 ms | lost 5–49 µs | | 25 ms | 18/18 lost 10–20 ms | 17/17 lost 10–20 ms | lost 5–49 µs | | 35 ms | 18/18 lost 20–30 ms | 17/17 lost 20–30 ms | lost 5–49 µs | **Drift, 6 concurrent guests over 300 s with a busy vCPU each:** - 8254 path: stock −0.045 to −1.584 s, fixed −0.005 s on all six. - LAPIC path: stock −0.168 to −3.287 s, fixed −0.019 to −0.030 s. `misc timer`, `vm clock` (no backward steps), `make check-misc` and `check-vm` pass on both paths. **Left open:** - The fix assumes a constant-rate TSC. A CPU whose TSC changes with clock speed would miscount reloads. - `hardclock`'s scheduler and statistics accounting still sees one call per interrupt; only the clock is corrected.