fix: close missed-wakeup race in FrameReceiver::recv() - #24
Open
SoundMatt wants to merge 1 commit into
Open
Conversation
FrameReceiver::recv() checked the queue and the closed flag, and only then constructed and awaited a Notify::notified() future. Bus::close() sets the closed flag and calls notify_waiters() in that order too, with no synchronization against recv()'s check-then-await sequence. notify_waiters() only wakes notified() futures that already exist (were already polling) at the moment it's called — per its own documented contract. If close() runs entirely between recv()'s closed-check and its construction of the notified() future, the wakeup fires into nothing: recv() then awaits a notified() future built after the fact, and hangs forever on an already-closed, already-empty bus. Fix by constructing the notified() future first, then checking the queue/closed state, per the check-then-await pattern Notify's own docs demonstrate — any close() that lands after the future is registered still wakes it; nothing narrower can slip through. Also add a concurrent regression test (multi-thread runtime, 500 trials, tokio::time::timeout) exercising this exact path: recv() pending on an *empty* queue (so it can't return via the pop() fast path) raced against a concurrent close(). The race window this closes is only a few CPU instructions wide, so it isn't reliably reproducible without genuine thread-level parallelism or delay injection — this test is a best-effort concurrent stress check, not a guaranteed reproduction of the pre-fix hang; the fix itself is correct by construction against Notify's documented contract independent of whether any given CI run's scheduling happens to land in the window. Closes #12. Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FrameReceiver::recv()checked the queue and theclosedflag, and only then constructed and awaited aNotify::notified()future.Bus::close()setsclosedand callsnotify_waiters()in that order, with no synchronization againstrecv()'s check-then-await sequence.notify_waiters()only wakesnotified()futures that already exist (are already registered/polling) at the moment it's called — per its own documented contract. Ifclose()runs entirely betweenrecv()'s closed-check and its construction of thenotified()future, the wakeup fires into nothing:recv()then awaits anotified()future built after the fact, and hangs forever on an already-closed, already-empty bus.FrameReceiver/SubInneris shared byvirtual_bus,mock,slave, andadapt.rs's subscribe loop, so any of them can hang a subscriber task on bus shutdown if the timing lands wrong.Fix
Construct the
notified()future first, then check the queue/closed state — matching the check-then-await patternNotify's own docs demonstrate. Anyclose()that lands after the future is registered still wakes it; nothing narrower can slip through.Also adds a concurrent regression test (multi-thread runtime, 500 trials,
tokio::time::timeout) exercising this exact path:recv()pending on an empty queue (so it can't return via thepop()fast path — the existingframe_receiver_recv_and_closetest pushes a frame first, so it never exercises the race path) raced against a concurrentclose().Note on test reliability: the race window this closes is only a few CPU instructions wide (between an atomic load and registering a waiter), so — as I confirmed empirically while writing this test — it is not reliably reproducible via black-box scheduling jitter alone, even across 500 trials on a multi-thread runtime; it would need delay injection or a tool like
loomto force deterministically. The new test is therefore a best-effort concurrent stress check, not a guaranteed reproduction of the pre-fix hang. The fix itself is correct by construction againstNotify's documented contract, independent of whether any given CI run's scheduling happens to land in the window.Closes #12.
Test plan
cargo fmt --checkcargo clippy --all-targets -- -D warningscargo test --locked(47 unit/integration + 2 doc tests, including the new regression test)