fix(pytorch_xdit): replace placeholder thresholds with measured values; fix cache paths - #324
Open
speriaswamy-amd wants to merge 2 commits into
Conversation
…s; fix cache paths
Thresholds. The shipped mi300x thresholds were unvalidated placeholders and one
was wrong by more than an order of magnitude: wan22_i2v_a14b set
max_avg_total_time_s: 12.0, while a real MI300X run with compile: true averages
161.03s/step (5 clean steps: 160.98, 161.03, 160.98, 161.06, 161.11). Every
real run on that hardware failed the threshold. Replaced with values derived
from live measurements, plus mi350 entries measured on Ruby:
wan22_i2v_a14b mi300x 12.0 -> 200.0 (161.03s/step measured, ~25% margin)
wan22_i2v_a14b mi350 new 120.0 (86.10-90.99s/step across two nodes)
flux1_dev_t2i mi300x 8.0 -> 3.0 (1.39s/pipe over 25 reps, ~2x margin)
flux1_dev_t2i mi350 new 2.0 (0.92s/pipe, consistent across 2 runs)
auto and mi355 are left as-is and remain unvalidated. Each config gains an
_expected_results_comment recording which numbers are measured, on what
hardware and with what margin, and which are not — so the next reader does not
have to guess again.
Schema. PytorchXditWan22Benchmarks and PytorchXditFlux1DevBenchmarks use
extra="forbid", so the comment field would make every test in the module fail
validation before running. Both switch to extra="allow", matching the other
schemas in the file that already permit comment fields.
Cache paths. hf_home was "/home/{user-id}", but HF_HOME must point at the root
containing hub/ — the test builds hub/models--*/snapshots/<rev> underneath it,
which does not exist directly in a home directory. Corrected to
"{home}/.cache/huggingface", the hf CLI default. hf_token_file and
output_base_dir move from "/home/{user-id}/..." to "{home}/...", which resolves
correctly where homes are not under /home.
Co-Authored-By: Claude <noreply@anthropic.com>
The _expected_results_comment fields named an internal AMD cluster; this repo is public, so keep the measurement basis without the cluster name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 11 of 12 in a stack that replaces #184. Base: #323.
Thresholds (AIMVT-88)
The shipped
mi300xthresholds were unvalidated placeholders and one was wrong by more than an order of magnitude:wan22_i2v_a14bsetmax_avg_total_time_s: 12.0, while a real MI300X run withcompile: trueaverages 161.03 s/step (5 clean steps: 160.98, 161.03, 160.98, 161.06, 161.11). Every real run on that hardware failed the threshold.wan22_i2v_a14bwan22_i2v_a14bflux1_dev_t2iflux1_dev_t2iautoandmi355are unchanged and remain unvalidated. Each config gains an_expected_results_commentrecording which numbers are measured, on what hardware, with what margin — and which are not.Schema
PytorchXditWan22BenchmarksandPytorchXditFlux1DevBenchmarksuseextra="forbid", so the comment field would make every test in the module fail validation before running. Both switch toextra="allow", matching the other schemas in the same file that already permit comment fields.Cache paths
hf_homewas/home/{user-id}, butHF_HOMEmust point at the root containinghub/— the test buildshub/models--*/snapshots/<rev>underneath it, which does not exist directly in a home directory. Corrected to{home}/.cache/huggingface, thehfCLI default.hf_token_fileandoutput_base_dirmove from/home/{user-id}/...to{home}/..., which resolves correctly where homes are not under/home.