Investigate the current DtoH sync solution in pipelines #13375

Activity

1. sayakpaul

   added

   performanceAnything related to performance improvements, profiling and benchmarkingAnything related to performance improvements, profiling and benchmarking

   on Apr 1, 2026
2. wunianze666-netizen commented on Sep 20, 2026

   

   wunianze666-netizen

   on Sep 20, 2026

   Contributor

   More actions

   I investigated a currently uncovered DtoH-sync path in Cosmos3OmniPipeline on current main (80c7ed262). The denoising path still has three independent scalar round-trips:

   • _mask_velocity_predictions() branches on mask.sum() > 0, which emits aten::_local_scalar_dense once per active modality and cond/uncond pass.
   • each scheduler timestep goes through t.item() and a new torch.full(...) before being sent back to the model device.
   • the first scheduler step searches for its timestep index even though this pipeline always starts at index zero.

   I prepared a focused patch at wunianze666-netizen@169ca79. It keeps masking tensor-native with torch.where (including the prior all-conditioned/NaN-to-zero behavior), sets the scheduler begin index after schedule construction, and expands the device timestep without copying it through Python.

   Executable evidence:

   • before the production change, the mask regression observed 3 aten::_local_scalar_dense calls; the scheduler-search guard failed; and expanded transformer timesteps did not share scheduler storage
   • after the change, both focused regressions pass
   • full tests/pipelines/cosmos/test_cosmos3.py: 22 passed, 20 skipped
   • Ruff check/format and git diff --check pass
   • self-review found no blocking, non-blocking, or dead-code findings

   I did not open another PR yet because this is a focused slice of the broader #13375 work. Would this Cosmos3-specific scope be useful as a PR?
3. wunianze666-netizen commented on Sep 21, 2026

   

   wunianze666-netizen

   on Sep 21, 2026

   Contributor

   More actions

   I found a second, independent #13375 slice in MiniMax-H3 and prepared a verified patch at wunianze666-netizen@be8f94790.

   MiniMaxH3SetTimestepsStep converted three device scalars to Python on every model-evaluation step, rebuilt the packed-row timestep map on CPU, then copied it back to the model device. Both schedulers also searched for their known first index and extracted it with .item().

   The patch keeps the computation device-native with a fixed four-role timestep table (generated video, generated audio, conditioned video, conditioned audio) and sets both scheduler begin indices to zero. Duplicate table entries are intentionally retained when roles share a timestep; the regression verifies that every packed row still selects exactly its previous value.

   Verification:

   • before: timestep setup emitted 9 aten::_local_scalar_dense calls for the 4-point test schedule
   • after: setup plus both schedulers' first steps emit 0 scalar extractions
   • T2VA and FL2VA generator-reproducibility tests pass
   • final focused run: 3 passed, 132 deselected
   • Ruff, formatting, git diff --check, secret/path scan, and project self-review pass (0 blocking, 0 non-blocking findings; READY)

   Per the AI-assisted contribution guide, I have not opened a PR without explicit maintainer acknowledgment. Would this MiniMax-H3 scope be welcome as a focused PR? I can keep the separate Cosmos3 slice from my earlier comment parked until its scope is also confirmed.