Drain result_queue before dispatch to prevent runtime deadlock in LocalExecutor - #70545
Open
SakshamKapoor2911 wants to merge 2 commits into
Open
Drain result_queue before dispatch to prevent runtime deadlock in LocalExecutor#70545SakshamKapoor2911 wants to merge 2 commits into
SakshamKapoor2911 wants to merge 2 commits into
Conversation
SakshamKapoor2911
requested review from
XD-DENG,
ashb,
dheerajturaga,
hussein-awala,
o-nikolas and
pierrejeambrun
as code owners
July 27, 2026 16:46
boring-cyborg
Bot
added
the
area:Executors-core
2 tasks
…alExecutor Call _read_results() before each activity_queue.put() in _process_workloads so workers blocked on a full result_queue pipe can resume reading from activity_queue, preventing the circular deadlock described in apache#70526. This is the dispatch-side equivalent of the fix applied to end() in apache#67881.
Verify _read_results() is called before each activity_queue.put() during dispatch, guarding against the dispatch-side deadlock described in apache#70526.
SakshamKapoor2911
force-pushed
the
fix-local-executor-dispatch-deadlock
branch
from
July 27, 2026 17:10
f03a04a to
df19a88
Compare
potiuk
added
the
ready for maintainer review
Contributor
Author
|
@eladkal @amoghrajesh — all CI checks pass. Fixes a LocalExecutor shutdown deadlock. Ready for maintainer review. |
eladkal
added
type:bug-fix
vatsrahul1001
approved these changes
Aug 4, 2026
Contributor
|
LGTM! should be good to merged post code owner review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a runtime deadlock in LocalExecutor where the scheduler blocks indefinitely on activity_queue.put() during task dispatch, a gap left by #67881 which only addressed the shutdown path.
Workers fill the result_queue pipe buffer and block on put(). Blocked workers stop consuming activity_queue, so the scheduler blocks on activity_queue.put() and never reaches _read_results() to drain the pipe — a circular deadlock.
The fix: drain result_queue via _read_results() before each activity_queue.put() in _process_workloads, using the same drain-before-write pattern already applied to end().
closes: #70526
Was generative AI tooling used to co-author this PR?
Generated-by: OpenCode (DeepSeek v4 Pro) following the guidelines