fix(pytorch_xdit): fail fast on missing image and on incomplete WAN caches - #321
Open
speriaswamy-amd wants to merge 1 commit into
Conversation
…aches Two preflight gaps, both of which turned a clear setup problem into a confusing downstream failure. Missing container image: neither benchmark checked that config.container_image was present locally before `docker run`. Both now call docker_lib.nodes_missing_docker_image() and fail with the node list and the `docker pull` command to fix it. WAN also gains the /dev/kfd check FLUX already had. Incomplete WAN snapshot: verification only ran `test -d` on the snapshot directory. That directory exists as soon as `hf download` starts, so a half-downloaded cache passed verification and the failure surfaced 20 minutes later inside the container. New utils_lib.wan_hf_snapshot_offline_check_commands() returns per-file OK/MISSING shell checks for the Wan2.2-I2V-A14B layout: the three config.json files, six >500MiB diffusion shards in each of the low/high-noise directories, and size floors on the VAE and T5 weights so LFS pointer stubs do not count as present. The checks use `find -L`. A hf_hub cache stores weights as symlinks into blobs/, so `find` without -L stats the link (tiny, type l) and reports zero shards for a fully correct cache. Verified against a real ~118GB download: without -L the low-noise shard count is 0, with -L it is 6. They run in both staging modes (explicit host path and hf_home cache) and only when the repo id or path names Wan2.2-I2V-A14B, so other WAN forks and custom layouts are unaffected. Adds a unit test asserting the generated commands quote paths containing spaces and emit both OK and MISSING branches. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 8 of 12 in a stack that replaces #184. Base: #320.
Why
Two preflight gaps, both of which turned a clear setup problem into a confusing downstream failure.
What changed
Missing container image. Neither benchmark checked that
config.container_imagewas present locally beforedocker run. Both now calldocker_lib.nodes_missing_docker_image()and fail with the node list and thedocker pullcommand to fix it. WAN also gains the/dev/kfdcheck FLUX already had.Incomplete WAN snapshot. Verification only ran
test -don the snapshot directory — which exists as soon ashf downloadstarts, so a half-downloaded cache passed and the failure surfaced ~20 minutes later inside the container. Newutils_lib.wan_hf_snapshot_offline_check_commands()returns per-file OK/MISSING shell checks for the Wan2.2-I2V-A14B layout: threeconfig.jsonfiles, six >500MiB diffusion shards in each of the low/high-noise dirs, and size floors on the VAE and T5 weights so LFS pointer stubs do not count as present.The checks use
find -L. Ahf_hubcache stores weights as symlinks intoblobs/, sofindwithout-Lstats the link (tiny, typel) and reports zero shards for a fully correct cache. Verified against a real ~118GB download: without-Lthe low-noise shard count is 0, with-Lit is 6.They run in both staging modes (explicit host path and
hf_homecache) and only when the repo id or path namesWan2.2-I2V-A14B, so other WAN forks and custom layouts are unaffected.Unit test asserts the generated commands quote paths containing spaces and emit both branches.