fix(recording): keep startup frame drops from shifting video ahead of audio

[richiemcilroy]

Every video frame dropped while a recording starts (capture pool still full while VideoToolbox initialises: 0 to 6 drops per run on main) shifted video one frame (16.7 ms at 60 fps) early relative to audio for the whole recording. Natural 60 s recordings on main measured a 44 ms median audio-video offset instead of the constant ~21 ms AAC priming.

Root cause

xychart-beta
  title "2 frames dropped at start, 60 fps: video PTS minus capture time (ms)"
  x-axis ["f0", "f3", "f4", "f5", "2 s anchor", "10 s", "60 s"]
  y-axis "ms" -40 --> 5
  line [0, -33, -33, -33, -33, -33, -33]
  line [0, -33, 0, 0, 0, 0, 0]

Loading

Upper line: main. Lower line: this PR (only the first frame after the skip stays on the snapped slot).

• crates/timestamp/src/master_clock.rs:294-309 snaps any frame within 70 ms of next_expected onto next_expected. After a skip the next frame is a whole slot late, and it is snapped back onto the skipped slot.
• master_clock.rs:318 derives the next expected time from the snapped value, so the ladder stays a slot behind for as long as frames keep arriving inside 70 ms. Up to 4 consecutive drops at 60 fps stick (5+ exceed the threshold and reset).
• crates/recording/src/output_pipeline/core.rs:1188 emits that lagged time for the first 2 s, core.rs:1196 takes the drift anchor from it, and core.rs:1206 emits anchor + wall clock afterwards, so the lag is frozen into the rest of the recording.

The snap already bounds how far the ladder may lead the source. Nothing bounded the lag.

Fix

• SourceClockState::detecting_skipped_frames() (opt in, used only by the video mux task on macOS, where screen and camera frames carry their CMSampleBuffer capture time; the audio mixer's remap is untouched). In the snap branch:
  • lag below half a frame: snapped exactly as before
  • lag of half a frame or more: recorded as pending and snapped as before
  • the next frame's lag has not shrunk by more than an eighth of a frame (the source kept its cadence, so slots were really skipped): the ladder moves onto the frame's real timestamp, and further whole-slot lags are taken directly until a frame lands back on the cadence
  • a backlog that drains (Windows cameras stamp frames on callback arrival, so a stall shows up as a lag that shrinks with each queued frame, even at 14 ms spacing for 60 fps) or a single late frame never confirms, so it is smoothed exactly as before
• VideoDriftTracker::defer_anchor: the 2 s anchor is not taken on a frame that is still pending, so a skip that lands on the anchor frame cannot freeze a lag either.
• Windows keeps the legacy remap. Its cameras stamp frames on callback arrival, so scheduling jitter of 5 ms or more can hold a frame half a slot late twice in a row and read as a skip: replaying arrival-stamped 60 fps traces (5 ms jitter, no drops) through the detector left 16 to 20 of 40 traces 4 to 15 ms off the capture clock where the legacy remap stayed exact.
• Linux uses remap_preserving_cadence and is unaffected.

let source_clock = SourceClockState::new("video");
#[cfg(target_os = "macos")]
let source_clock = source_clock.detecting_skipped_frames();
...
drift_tracker.defer_anchor(source_clock.lag_pending());

Results

Synthetic recorder driving the real macOS output pipelines (3024x1964 NV12 60 fps through VideoToolbox, mic + system audio, 1 s flash/click markers), base 68d6c31b4 vs this PR, interleaved A/B on the same host. Offset is audio minus video per marker; ~21 ms is the AAC priming constant (instant mode has none, so ~0 there).

class	runs per side	main: median offset (run medians)	PR: median offset (run medians)	main worst marker	PR worst marker
0 forced start-up drops	3	20.0 (19.9..86.3)	20.9 (19.5..23.0)	90.2	26.7
1 forced	3	36.9 (35.0..37.8)	22.4 (21.0..22.6)	42.7	37.7
2 forced	3	53.5 (53.1..57.5)	21.0 (20.1..21.4)	61.7	25.7
4 forced	3	84.6 (84.0..87.7)	21.0 (19.4..23.2)	92.9	28.0
5 forced	3	23.5 (23.2..54.3)	21.0 (17.5..21.1)	57.5	25.7
8 forced	3	20.9 (20.4..21.0)	20.6 (19.9..22.5)	87.7	27.1
natural 60 s (1-2 start-up drops each)	8	44.3 (33.1..54.5)	21.1 (17.5..21.9)	59.2	26.7
CPU contention (yes x8), 30 s	4	46.7 (17.9..86.9)	21.7 (21.1..22.0)	91.1	26.7
mid-recording drops incl. one on the anchor frame	3	53.7 (50.9..56.1)	22.6 (16.7..24.4)	59.7	28.3
instant mode, 2 start-up drops	3	33.3 (33.2..34.0)	0.7 (-1.7..0.9)	38.1	6.0
camera track (30 fps), 2 drops	3	86.0 (83.3..89.7)	21.0 (19.0..23.2)	n/a	n/a

Real capture, cap selftest av-sync (on-screen flashes and speaker beeps recorded through ScreenCaptureKit and system audio, Studio mode, 20 s, alternating runs). On a 120 Hz display ScreenCaptureKit delivers most frames 25 ms apart, so the 60 fps ladder sees half-slot lags all the time:

	main	this PR
measured A/V offset per run (ms)	+10, -7, +29, +59, +7, +36	-9, -28, -19, -19, -17, -19
spread across runs	66 ms	19 ms
video PTS minus capture time, run median (ms)	-33, -25	-4, 0
frames sent / packets in file	1080 / 1082	1054 / 1056
duplicate or non-monotonic PTS, decode errors	0	0

main freezes video two to three slots ahead of its capture time for most of each recording; this PR keeps it within half a slot, so the offset stops depending on how many frames dropped at start.

Runs on main with 0, 5 or 8 forced drops still lagged whenever the busy host caused natural drops inside the first 2 s (the 86.3 and 54.3 outliers).

No-skip behaviour is unchanged:

• Every frame of every run without drops produced the same pre-anchor PTS sequence on both sides (identical hash over the first 1.9 s).
• Classes that exercise jitter and stalls but never skip a slot (7 ms one-sided timestamp jitter, arrival-stamped screen and camera with 40 to 60 ms stalls, instant pause/resume) never take the new path (skip_count 0), so their remapped timestamps match main frame for frame. Muxed frame counts, durations, decode errors (0) and manifest completeness match across all classes.
• Replay tests assert the skip-tracking remap equals the legacy remap for jitter under half a frame, single late frames, draining backlogs, fast sources and 5+ frame skips, and that the full mux chain is identical without skips.

Confidence Score: 5/5

The PR appears safe to merge.

Summary

Corrects startup frame-drop timing and keeps the drift anchor off frames whose lag is still pending.

• Skipped video frames no longer pull later timestamps ahead of audio.

_{Reviews (3) · Last reviewed commit: "docs(timestamp): note skipped-frame dete..." · Reviewed by Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
cap-status	Ready	Preview	Oct 14, 2026 3:02pm UTC

[capy-ai]

Capy found no issues.

View 2 other findings in Capy.

[richiemcilroy]

hey @greptileai, please re-review the PR

[capy-ai]

Capy found no issues.

[richiemcilroy]

hey @greptileai, please re-review the PR

[capy-ai]

Capy found no issues.