Skip to content

fix(recording): keep audio through multi-second pipeline stalls - #2558

Open
sloemo01 wants to merge 3 commits into
CapSoftware:mainfrom
sloemo01:fix/recording-stall-resilience
Open

sloemo01 wants to merge 3 commits into
CapSoftware:mainfrom
sloemo01:fix/recording-stall-resilience

Conversation

@sloemo01

@sloemo01 sloemo01 commented Oct 14, 2026 •

Copy link
Copy Markdown

This picks up the audio side of #2366.

I can reproduce the stall pattern on a 4 GB SD card (~4.9 MB/s writes, exFAT) on an M3 Pro. An 11.4 min Studio recording at 3600x2338/60 with the software encoder (VideoToolbox zero-copy input unavailable at that size):

  • 34 wall-clock-confirmed delivery gaps, 2.2 to 5.2 s each, 115.4 s of video that never arrived (16.8% of the recording)
  • the frame-timeline gap matches the wall-clock gap to the millisecond, so frames genuinely stopped arriving from the capture side, and the muxer's 5% drop-rate warning never fired; nothing in Cap dropped them
  • recordings to the internal disk on the same machine show zero gaps, on every prior day's log; the card is what makes it happen

Initial cadence was ~30 s, tightening over the recording, and the first gap hit 6 s in. Same shape as the report, minus the audio drops, because nothing was playing in my run.

What this changes (crates/recording):

  1. Audio channels go from 128 to 2048 frames: ~40 s of 20 ms frames, a few tens of MB per channel. At 128 frames, any consumer stall longer than ~2.7 s sheds frames, and the muxer fills the hole with silence. Buffering audio is cheap; the video path already holds ~300 frames of 3600x2338 and cannot be buffered any deeper for free.
  2. The stop path drains up to the channel capacity instead of a fixed 500 frames, so a stall that ends with a long backlog still gets written out. Main added a stop deadline (MIXER_STOP_TIMEOUT) while this was open; the rebase keeps that bound and only raises the frame count.
  3. The system-audio stats task now warns only when the drop counter advanced, instead of warning every 5 s for the rest of the recording after a single drop.
  4. Microphone send timeout goes from 5 ms wired / 20 ms wireless to 50 ms / 200 ms. The old budget was small enough that scheduler jitter dropped frames even when the channel had room; with the channel absorbing long stalls, the timeout only needs to catch a stuck feed.

Not addressed: frames the capture stack never delivered during a stall cannot be recovered in the app. The timestamp tracker already logs each confirmed gap and keeps the gap in the timeline so video stays in sync with audio.

Verification on the rebased head (49a8523):

  • cargo check -p cap-recording and cargo fmt --all --check clean.
  • cargo test -p cap-recording --lib: 690 passed, 0 failed.
  • The Linux-gated tests that fill the production audio queue do not compile on macOS, so ubuntu CI is their only verifier.

Buffer sizing is easy to change if you would rather spend memory differently.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no blocking issues were found.

Summary

The PR gives audio more room to survive slow output:

  • Recording audio queues hold more frames through pipeline stalls.
  • macOS system-audio warnings appear only when new frames drop.

No new actionable issues were found. Both earlier, unnumbered findings were checked without reposting them. The current code removes the 500-frame drain limit and replaces the separate byte-queue test with real audio frames through the production helper.

Review checks were static; tests were not run.

Reviews (6) · Last reviewed commit: "test(recording): fill the combined audio..." · Reviewed by Greptile

@vercel

vercel Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

@sloemo01 is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

Comment thread crates/recording/src/output_pipeline/core.rs Outdated
Comment thread crates/recording/src/output_pipeline/core.rs Outdated
@richiemcilroy

Copy link
Copy Markdown
Member

Thanks for the PR. Greptile currently has this at 4/5, and we only merge PRs once Greptile scores the current head 5/5. Please address the remaining Greptile feedback, then comment @greptileai to request a re-review, and repeat until it reaches 5/5.

@sloemo01
sloemo01 force-pushed the fix/recording-stall-resilience branch from c0acbbf to 266cf87 Compare October 14, 2026 06:52
@sloemo01

Copy link
Copy Markdown
Author

@greptileai

1 similar comment
@sloemo01

Copy link
Copy Markdown
Author

@greptileai

@sloemo01

Copy link
Copy Markdown
Author

@richiemcilroy the ubuntu sync failure is fixed: a Linux-only test filled the audio channel with a fixed 256 sends and the channel is now 2048 frames, so the loop follows the capacity constant instead. It doesn't compile on macOS, which is why local runs never saw it. Greptile has the new head at 5/5 and the re-run is queued at the approval gate. Ready for another look when you get a chance.

@sloemo01
sloemo01 force-pushed the fix/recording-stall-resilience branch from 8d3aee7 to 735d65b Compare October 14, 2026 08:00
@sloemo01

Copy link
Copy Markdown
Author

@greptileai

@sloemo01

Copy link
Copy Markdown
Author

@richiemcilroy the previous Windows failure was the stale-base quirk in that job (it reads win.rs at the PR's stored base commit), so I rebased onto current main. Greptile has the new head at 5/5, and the fresh runs are queued waiting on approval. Ready for another look once they go through.

@sloemo01

Copy link
Copy Markdown
Author

@richiemcilroy checking in on this one: the four fresh runs from this morning are still waiting on approval, so CI has not run on the rebased head yet. Greptile is already 5/5 on the new head and the rebase fix is in, so approving them should get this over the line.

@sloemo01

Copy link
Copy Markdown
Author

@richiemcilroy small nudge: the four queued runs are still waiting on approval, so CI has not run on the rebased head yet. Greptile is 5/5 on that commit and the rebase fix is in, so approving them is the only thing left on this one.

@richiemcilroy

Copy link
Copy Markdown
Member

Thanks for this! It now conflicts with main after today's merges. Could you merge or rebase onto the latest main? Once it's up to date I'll run the full engine verify and real-capture suites on it and merge if they pass.

Audio channels were 128 frames deep (~2.7 s of 20 ms frames), so a
consumer stall longer than that made sources shed frames and the muxer
later filled the hole with silence. Recording to slow storage can stall
consumers for seconds at a time (CapSoftware#2366). 2048 frames is ~40 s of audio
and costs a few tens of MB, small next to what the video pipeline
already holds.

The system-audio stats task also warned every 5 s for the rest of a
recording after a single drop; now it warns only when the drop counter
advanced. The microphone send timeout was tight enough (5 ms wired,
20 ms wireless) for scheduler jitter to drop frames even with channel
space available, so it moves to 50 ms / 200 ms; the channel itself now
absorbs the multi-second stalls.
The stop path only saved 500 queued audio frames, so with the enlarged source queue a long stall could still end with discarded samples and silence padding. The drain now runs to the channel capacity.

The stall test goes through the production channel factory with real 20 ms frames instead of a private byte queue, so it fails if a production queue shrinks back.
…pacity

The drain-failure test assumed a 128-frame source channel and capped its
fill loop at 256 sends. The source channel now buffers
AUDIO_SOURCE_CHANNEL_CAPACITY frames, so the loop bound follows the
constant instead of a hardcoded count.
@sloemo01
sloemo01 force-pushed the fix/recording-stall-resilience branch from 735d65b to 49a8523 Compare October 19, 2026 17:28
@sloemo01

Copy link
Copy Markdown
Author

Rebased onto current main, head is 49a8523.

One real conflict, in the stop drain: main had added MIXER_STOP_TIMEOUT plus a 500-frame cap while this was open. I kept the deadline and let the frame count follow AUDIO_SOURCE_CHANNEL_CAPACITY, which is the part this PR needs. Nothing else conflicted.

Local checks on the new head: cargo check -p cap-recording clean, cargo test -p cap-recording --lib 690 passed, cargo fmt --all --check clean. The queue-filling test that guards this is Linux-gated, so ubuntu CI is the only place it runs and I could not run it locally.

The new runs are parked at the approval gate, so they need a click before CI actually runs on this head. Greptile re-review is requested. Ready for the engine verify and real-capture suites once those get through.

@sloemo01

Copy link
Copy Markdown
Author

@greptileai

1 similar comment
@sloemo01

Copy link
Copy Markdown
Author

@greptileai

@sloemo01

Copy link
Copy Markdown
Author

@richiemcilroy rebased onto today's main, head is 49a8523, Greptile 5/5 on it. Local checks are clean. The new runs are sitting at the approval gate, so the engine verify and real-capture suites need a click before they start.

@greptile-apps

greptile-apps Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

@sloemo01

Copy link
Copy Markdown
Author

You asked for the rebase after the conflicts, that's done on 49a8523 and Greptile is 5/5 on it. The engine verify and real-capture runs are parked at the approval gate, so they're ready to go whenever you click them through.

@sloemo01

Copy link
Copy Markdown
Author

Quick nudge: main moved three times overnight and 49a8523 still merges clean, Greptile 5/5 on that head. The seven checks are all parked at the approval gate, so the recording reliability and A/V sync suites haven't run on it yet. One click and they start.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants