🐛 fix(venv): insidious creation/cleanup race condition#940
Open
13steinj wants to merge 1 commit intoaspect-build:mainfrom
Open
🐛 fix(venv): insidious creation/cleanup race condition#94013steinj wants to merge 1 commit intoaspect-build:mainfrom
13steinj wants to merge 1 commit intoaspect-build:mainfrom
Conversation
|
|
|
Suffixes the venv created with the running shell's PID, ensuring that each call generates a venv at a unique location. Also, explicitly cleans the venv up at the end to avoid too many piling up. Fixes aspect-build#858, aspect-build#899.
13steinj
force-pushed
the
fix-venv-race-condition
branch
from
cca2ab3 to
b86722f
Compare
13steinj
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Suffixes the venv created with the running shell's PID, ensuring that each call generates a venv at a unique location.
Also, explicitly cleans the venv up at the end to avoid too many piling up.
Fixes #858, #899.
I am near certain the issue I've been wrestling with is the same as #858 and that the root cause that this PR fixes is the same as the root cause for #899. I don't know if there is any code path in which the venv can be created outside the bazel output base, which is why I added the
trapcleanup. If there isn't I am happy to remove that line.Test plan
py_binaryand specificvenvs (say, via the unstableuvbased rules anddependency_groupsand auv.locklockfile, and more specifically, one that has build actions that use apy_binaryas a code generator), attempt to build on a beefy 96 core machine with just the right actions cached such that the venvs collide.Sadly as this is a race condition your machine needs to be just beefy enough that the actions run in the right order to create the perfect storm of pain.
Then, do it again with this branch, and see that the venvs no longer collide.
If one is using remote execution, you will not (or at least it will be very difficult) to be able to reproduce the original bug, because it appears that under remote execution the venvs are (a) potentially sharded across different nodes, and (b) I imagine the remote execution implementation defines its own prefix path per action, such that I am sure that there exist implementations that have a more unique path based on the action information.