Skip to content

fix(recording): keep macOS recordings intact through output storage stalls - #2566

Open
richiemcilroy wants to merge 4 commits into
mainfrom
fix/recording-storage-stall-write-behind
Open

richiemcilroy wants to merge 4 commits into
mainfrom
fix/recording-storage-stall-write-behind

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Fixes #2366

When the recording volume stops accepting writes for a while (slow SD card, external or network drive), macOS Studio and Instant recordings lost every video frame and most audio for the length of the stall. This moves file writes for those outputs onto per-output writer threads with a bounded in-memory buffer, so capture and encoding keep running while storage catches up.

flowchart LR
  subgraph Before
    A1["ScreenCaptureKit frames"] --> B1["m4s encoder thread: VideoToolbox encode + DASH write + manifest fsync"]
    C1["mic / system audio"] --> D1["tokio worker (mux-audio): AAC encode + write()"]
    B1 -- "blocks in write()" --> V1[("output volume")]
    D1 -- "blocks in write()" --> V1
  end
  subgraph After
    A2["ScreenCaptureKit frames"] --> B2["m4s encoder thread: VideoToolbox encode"]
    B2 -- "packets" --> Q2[["byte-budgeted queue"]]
    Q2 --> W2["mux-writer-video: DASH write, segment events, manifests"]
    C2["mic / system audio"] --> D2["tokio worker (mux-audio): AAC encode"]
    D2 -- "packets" --> Q3[["byte-budgeted queue"]]
    Q3 --> W3["mux-writer-audio"]
    W2 --> V2[("output volume")]
    W3 --> V2
  end
Loading

Root cause

  • Video: the encoder thread encodes and writes on the same call path (crates/enc-ffmpeg/src/mux/segmented_stream.rs:380, called from crates/recording/src/output_pipeline/macos_fragmented_m4s.rs:417), and every segment boundary also writes manifest.json with two fsyncs (segmented_stream.rs:29-42). While a write blocks, the sample buffers already handed to the encoder channel (macos_fragmented_m4s.rs:287, filled at :511) stay retained, the capture surface pool (queue depth 8, crates/recording/src/sources/screen_capture/macos.rs:240) runs dry, and capture stops delivering. That is the "Wall-clock-confirmed forward jump" in the issue log.
  • Audio: FragmentedAudioMuxer::send_audio_frame (crates/recording/src/output_pipeline/ffmpeg.rs:235) writes the file inline (crates/enc-ffmpeg/src/mux/fragmented_audio.rs:100) from the mux-audio task (crates/recording/src/output_pipeline/core.rs:3380, :4037). Each stalled audio track pins one tokio worker, the source channels fill, and the sources drop frames ("dropping frames due to full channel" plus inserted silence). With few free workers the runtime itself stops making progress.
  • Stopping during a stall waited 5 s on the blocked encoder thread, then failed with the video track truncated at the start of the stall.

Change

  • crates/enc-ffmpeg/src/mux/write_behind.rs: a writer thread per output owns the AVFormatContext. Encoders hand packets through the existing PacketSink trait (new queue_frame_to / flush_to on the H264, AAC and Opus encoders). Packets are written in production order, so the bytes on disk are the same as inline writing.
  • DASH segment detection, segment events and manifest.json writes run on the writer thread, including the initial manifest, so the first-frame path no longer fsyncs.
  • Budget: video gets 40 s at the encoder target bitrate, clamped to 32 MiB minimum and to the smaller of 512 MiB or RAM / 32 (256 MiB on an 8 GB Mac). Each audio output gets 16 MiB. When a budget is full the producer waits for storage, which is the old behaviour, and logs it once.
  • macOS only: the write_behind flag on OggMuxerConfig, FragmentedAudioMuxerConfig and DashSegmentedAudioMuxerConfig defaults to off and is set with cfg!(target_os = "macos") at the Studio and Instant call sites. Video uses SegmentedVideoEncoder::init_write_behind from the macOS-only M4S muxers. Windows and Linux keep the inline write path.
  • OggMuxer now takes OggMuxerConfig instead of ().
  • Stop is one bounded operation of at most 4 s, run by the writer thread:
    • queued packets drain first; if storage is still accepting writes after 3 s, the unwritten tail is dropped;
    • the writer then writes the trailer and final manifest itself, inside the same 4 s;
    • if that has not completed in 4 s, finish returns an error, finalization is cancelled, and the files stay in the state recovery expects. Queued packets still reach disk once storage recovers, as they would from an encoder blocked in an inline write.
  • Segment callbacks are installed without waiting on the writer thread, so a stalled initial manifest write never blocks the first frame.
  • Dropping a writer that was never finished discards its queue, joins for at most 250 ms, and writes nothing after its current call returns.
  • Errors: per-packet write errors are logged on the writer thread (rate-limited) and never abort the end-of-stream flush. Only failures on the flush packets fail finish, matching an inline flush. A writer panic is reported on the next enqueue.

Results

Stalls were reproduced with a DYLD_INSERT_LIBRARIES interposer that blocks write, pwrite, writev, fsync, F_FULLFSYNC, write-mode open and rename under the recording directory, driving a synthetic pipeline: 3024x1964 NV12 at 60 fps through the real VideoToolbox encoder and M4S muxer (capture pool of 8 buffers), plus 48 kHz mic and system audio through the real audio muxers. Each class ran as interleaved pairs, with main as the baseline. The A/V offset is measured from flash and click markers in the output files and includes a constant 21 ms AAC priming offset.

180 s with a 20 s stall every 60 s (4 runs each)

main this PR
Video frames in the file (about 10,800 captured) 7,114 avg 10,771 avg
Frames dropped at the capture pool 14,630 0
Mic / system audio chunks dropped 3,512 / 8,043 0 / 0
A/V offset, median (range) 44.8 ms (32.8 to 54.8) 21.9 ms (20.0 to 23.2)
Finalized files 4 / 4 4 / 4
Peak memory 612 MB 944 MB

Stop 12.5 s into a 20 s stall (4 runs each)

main this PR
Stop time 5.00 to 5.04 s 4.03 to 4.08 s
Video on disk afterwards (42 s recorded) 29.6 to 30.4 s 40.0 to 40.2 s
Frames dropped at the capture pool, per run about 690 0
Mic / system audio chunks dropped, per run 67 / 299 0 / 0
Final manifest written no (left for recovery) no (left for recovery)

15 s recordings, no stall (20 runs main, 19 this PR)

main this PR
Runs with dropped frames at start-up (frames) 12 (19) 6 (14)
Dropped frames after start-up 5 0
A/V offset, median (range) 35.8 ms (17.8 to 71.3) 21.0 ms (17.9 to 38.4)
Encoder init to first frame accepted 21.9 ms 0.1 ms

60 s recordings, no stall (12 runs each)

main this PR
Video frames in the file 3,570 avg 3,571 avg
Runs with dropped frames at start-up (frames) 4 (7) 0
Dropped frames after start-up 0 1
A/V offset, median (range) 22.9 ms (17.7 to 71.0) 21.2 ms (18.9 to 24.1)
Peak memory, max 674 MB 682 MB

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no blocking issues remain.

Summary

The PR moves macOS recording writes onto dedicated threads with bounded packet buffers. The latest changes address both earlier findings:

  • macOS video encoding keeps moving during storage stalls.
  • macOS audio encoding keeps moving during storage stalls.

Both earlier threads were manually resolved by richiemcilroy without explanatory replies. The current code addresses their reported causes. Neither thread supplied a Comment number, so neither appears in previousFindings.

No new actionable issues were found. Checks were based on code inspection; no builds or runtime tests were run.

Reviews (2) · Last reviewed commit: "fix(enc-ffmpeg): bound write-behind fina..." · Reviewed by Greptile

Comment thread crates/enc-ffmpeg/src/mux/segmented_stream.rs Outdated
Comment thread crates/enc-ffmpeg/src/mux/write_behind.rs Outdated

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 2 potential issues (2 medium).

View 2 other findings in Capy.

Open in Capy

Comment thread crates/enc-ffmpeg/src/mux/write_behind.rs Outdated
Comment thread crates/enc-ffmpeg/src/mux/segmented_stream.rs Outdated
@vercel

vercel Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
cap-status Ready Ready Preview Oct 14, 2026 10:09am UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

This branch was successfully deployed

2 active deployments
Preview – cap-web — 0db83e2f Deployed Oct 14, 2026 by vercel[bot]
Preview – cap-status — 0db83e2f Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Macos: Recording pipeline repeatedly stalls for 5–22 s (whole process), causing ~10% video loss and multi-second audio gaps

1 participant