Use os.posix_spawn instead of fork+exec in the task-sdk supervisor - #73503
Merged
kaxil merged 2 commits intoSep 22, 2026
Merged
Conversation
ashb
approved these changes
Sep 21, 2026
kaxil
reviewed
Sep 21, 2026
kaxil
approved these changes
Sep 21, 2026
execute_tasks_new_python_interpreter (apache#72164) and the macOS-forced exec path both still call os.fork() before execv(), and CPython's os.fork() runs every os.register_at_fork(after_in_child=...) callback synchronously inside the fork() call itself, before any Python-level code -- including the planned execv() -- gets control back. A third-party library's own fork handler that blocks there (confirmed live: a customer's task process hung inside datadog's dogstatsd client, which registers such a handler by default) hangs the child before exec is ever reached, regardless of how soon the caller tries to exec. This isn't just Airflow's own known OpenSSL provider-store case (apache#71707) -- it's any library that registers an at-fork handler that isn't async-signal-safe, which fork+exec cannot protect against structurally, no matter how the call sites are ordered. os.posix_spawn() doesn't have this gap: CPython's binding never calls PyOS_AfterFork_Child(), and glibc's own posix_spawn (2.24+) uses clone(CLONE_VM|CLONE_VFORK) rather than fork(), so registered os.register_at_fork()/pthread_atfork() handlers are structurally unreachable, not just less likely to hang. It's also not a new cost on top of the existing exec path -- benchmarked against a real Airflow import, posix_spawn is measurably not more expensive than the fork+exec it replaces (slightly cheaper, from skipping fork()'s own copy-on-write setup before the exec). No new config surface: wherever use_exec was already True (the platform gate or execute_tasks_new_python_interpreter), the spawn mechanism underneath is now always posix_spawn. That decision was already made by existing config; this only changes how "give me a fresh interpreter" is implemented once it's been decided, matching the existing dup2/env/ process-group semantics via posix_spawn's file_actions/env/setpgroup parameters instead of imperative code in a forked child.
kaxil
force-pushed
the
seanmuth/posix-spawn-instead-of-fork-exec
branch
from
September 21, 2026 21:11
7ff323a to
1b987ba
Compare
Fixes docstrings left describing the old os.fork()+os.execv() mechanism after it was replaced with os.posix_spawn(): the module docstring's os.set_inheritable/FD_CLOEXEC description, _child_exec_main's "placed there via dup2" description, and the dup2-ordering safety invariant that was dropped along with the old imperative dup2 code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
execute_tasks_new_python_interpreter(#72164) and the macOS-forced exec path both still callos.fork()beforeexecv(). CPython'sos.fork()runs everyos.register_at_fork(after_in_child=...)callback synchronously, inside thefork()call itself, before any Python-level code — including the plannedexecv()— gets control back. A third-party library's own fork handler that blocks there hangs the child before exec is ever reached, regardless of how soon the caller tries to exec — confirmed live: a deployment's task process hung insidedatadog's dogstatsd client, which registers exactly such a handler by default. This isn't just Airflow's own known OpenSSL provider-store case (#71707) — it's any library that registers an at-fork handler that isn't async-signal-safe, and fork+exec can't protect against that structurally, no matter how the call sites are ordered.os.posix_spawn()doesn't have this gap: CPython's binding never callsPyOS_AfterFork_Child(), and glibc's ownposix_spawn(2.24+) usesclone(CLONE_VM|CLONE_VFORK)rather thanfork(), so registeredos.register_at_fork()/pthread_atfork()handlers are structurally unreachable, not just less likely to hang. It's also not a new cost on top of the existing exec path — benchmarked against a real Airflow import,posix_spawnis measurably not more expensive than the fork+exec it replaces (slightly cheaper, from skippingfork()'s own copy-on-write setup before the exec).No new config surface: wherever
use_execwas alreadyTrue(the platform gate orexecute_tasks_new_python_interpreter), the spawn mechanism underneath is now alwaysposix_spawn. That decision is already made by existing config; this only changes how "give me a fresh interpreter" is implemented once it's been decided.related: #71707, #72164, #72493
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Sonnet 5 following the guidelines