feat(docker_lib): add image-presence preflight and a prune opt-out - #319
Open
speriaswamy-amd wants to merge 3 commits into
Open
Conversation
Two small additions needed by the pytorch_xdit benchmarks, both usable by any suite: nodes_missing_docker_image(phdl, image_ref) runs `docker image inspect` on every target and returns the hosts where the image is absent. Without it, a missing image surfaces as a `docker run` failure minutes into a test, mixed in with container output; callers can now fail fast with an actionable message. The image ref is shlex-quoted. delete_all_containers_and_volumes() now honours CVS_PYTORCH_XDIT_SKIP_DOCKER_SYSTEM_PRUNE (1/true/yes/on). The function runs `docker system prune --force`, which removes stopped containers and unused networks belonging to other users on shared cluster nodes. The opt-out lets a run stop its own named container and skip the cluster-wide prune. Default behaviour is unchanged. Adds unit tests for both, covering the missing/present cases and the skip path. Co-Authored-By: Claude <noreply@anthropic.com>
…it-06-docker-image-preflight (bring branch up to date)
amd-droy
reviewed
Aug 18, 2026
|
|
||
| This runs ``docker system prune --force`` (not ``-a``), but it can still remove | ||
| stopped containers and unused networks on shared nodes. Set environment variable | ||
| ``CVS_PYTORCH_XDIT_SKIP_DOCKER_SYSTEM_PRUNE`` to ``1``/``true``/``yes`` to skip this |
Contributor
There was a problem hiding this comment.
isn't this function used by most of the frameworks, can this 'CVS_PYTORCH_XDIT_SKIP_DOCKER_SYSTEM_PRUNE' be generalized to 'CVS_SKIP_DOCKER_SYSTEM_PRUNE'
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 6 of 12 in a stack that replaces #184. Base: #318.
Why
Two small library gaps the pytorch_xdit benchmarks hit, both usable by any suite.
What changed
nodes_missing_docker_image(phdl, image_ref)— runsdocker image inspecton every target and returns the hosts where the image is absent. Without it, a missing image surfaces as adocker runfailure minutes into a test, buried in container output. The image ref isshlex-quoted.delete_all_containers_and_volumes()now honoursCVS_PYTORCH_XDIT_SKIP_DOCKER_SYSTEM_PRUNE(1/true/yes/on). The function runsdocker system prune --force, which removes stopped containers and unused networks belonging to other users on shared cluster nodes. The opt-out lets a run stop its own named container and skip the cluster-wide prune. Default behaviour is unchanged.Unit tests cover both, including the missing/present cases and the skip path.