Skip to content

fix: close missed-wakeup race in FrameReceiver::recv() - #24

Open
SoundMatt wants to merge 1 commit into
mainfrom
fix/frame-receiver-missed-wakeup
Open

fix: close missed-wakeup race in FrameReceiver::recv()#24
SoundMatt wants to merge 1 commit into
mainfrom
fix/frame-receiver-missed-wakeup

Conversation

@SoundMatt

Copy link
Copy Markdown
Owner

Summary

FrameReceiver::recv() checked the queue and the closed flag, and only then constructed and awaited a Notify::notified() future. Bus::close() sets closed and calls notify_waiters() in that order, with no synchronization against recv()'s check-then-await sequence.

notify_waiters() only wakes notified() futures that already exist (are already registered/polling) at the moment it's called — per its own documented contract. If close() runs entirely between recv()'s closed-check and its construction of the notified() future, the wakeup fires into nothing: recv() then awaits a notified() future built after the fact, and hangs forever on an already-closed, already-empty bus. FrameReceiver/SubInner is shared by virtual_bus, mock, slave, and adapt.rs's subscribe loop, so any of them can hang a subscriber task on bus shutdown if the timing lands wrong.

Fix

Construct the notified() future first, then check the queue/closed state — matching the check-then-await pattern Notify's own docs demonstrate. Any close() that lands after the future is registered still wakes it; nothing narrower can slip through.

Also adds a concurrent regression test (multi-thread runtime, 500 trials, tokio::time::timeout) exercising this exact path: recv() pending on an empty queue (so it can't return via the pop() fast path — the existing frame_receiver_recv_and_close test pushes a frame first, so it never exercises the race path) raced against a concurrent close().

Note on test reliability: the race window this closes is only a few CPU instructions wide (between an atomic load and registering a waiter), so — as I confirmed empirically while writing this test — it is not reliably reproducible via black-box scheduling jitter alone, even across 500 trials on a multi-thread runtime; it would need delay injection or a tool like loom to force deterministically. The new test is therefore a best-effort concurrent stress check, not a guaranteed reproduction of the pre-fix hang. The fix itself is correct by construction against Notify's documented contract, independent of whether any given CI run's scheduling happens to land in the window.

Closes #12.

Test plan

  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test --locked (47 unit/integration + 2 doc tests, including the new regression test)
  • Verified the new test passes with the fix and does not itself introduce flakiness (ran repeatedly)
  • CI (will poll after opening)

FrameReceiver::recv() checked the queue and the closed flag, and only
then constructed and awaited a Notify::notified() future. Bus::close()
sets the closed flag and calls notify_waiters() in that order too, with
no synchronization against recv()'s check-then-await sequence.

notify_waiters() only wakes notified() futures that already exist (were
already polling) at the moment it's called — per its own documented
contract. If close() runs entirely between recv()'s closed-check and its
construction of the notified() future, the wakeup fires into nothing:
recv() then awaits a notified() future built after the fact, and hangs
forever on an already-closed, already-empty bus.

Fix by constructing the notified() future first, then checking the
queue/closed state, per the check-then-await pattern Notify's own docs
demonstrate — any close() that lands after the future is registered
still wakes it; nothing narrower can slip through.

Also add a concurrent regression test (multi-thread runtime, 500
trials, tokio::time::timeout) exercising this exact path: recv()
pending on an *empty* queue (so it can't return via the pop() fast
path) raced against a concurrent close(). The race window this closes
is only a few CPU instructions wide, so it isn't reliably reproducible
without genuine thread-level parallelism or delay injection — this
test is a best-effort concurrent stress check, not a guaranteed
reproduction of the pre-fix hang; the fix itself is correct by
construction against Notify's documented contract independent of
whether any given CI run's scheduling happens to land in the window.

Closes #12.

Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bus: missed-wakeup race in FrameReceiver::recv() can hang subscribers forever on close()

1 participant