Skip to content

fix(minimax-h3): align Ref2AV output #1490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Watebear wants to merge 15 commits into
base: main
Choose a base branch
from
+2,894 −584
Open

fix(minimax-h3): align Ref2AV output #1490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
68f453d
feat(minimax-h3): align Ref2AV output with SGLang
Watebear Sep 6, 2026
94a79ed
refactor(minimax-h3): localize SGLang parity ops
Watebear Sep 6, 2026
e202aeb
chore(minimax-h3): omit redundant config defaults
Watebear Sep 7, 2026
d650558
refactor(minimax-h3): unify SGL alignment config
Watebear Sep 7, 2026
90b579c
chore(minimax-h3): remove standalone parity tests
Watebear Sep 7, 2026
f01b303
fix(minimax-h3): support parity-safe model offload
Watebear Sep 7, 2026
1d163d7
refactor: simplify attention validation and parity guards
Watebear Sep 7, 2026
20fefd0
refactor(minimax-h3): align inference with SGL
Watebear Sep 9, 2026
221f1ba
refactor(minimax-h3): make aligned execution canonical
Watebear Sep 9, 2026
7a66c1b
refactor(minimax-h3): register configurable SGL RoPE
Watebear Sep 9, 2026
30682e2
refactor(minimax-h3): clarify weight and kernel boundaries
Watebear Sep 9, 2026
244ff9c
refactor(minimax-h3): generalize exact NeoX RoPE
Watebear Sep 9, 2026
13cd227
refactor(minimax-h3): generalize exact RMSNorm backend
Watebear Sep 9, 2026
e6ff9ae
fix(attn): keep non-packed SDPA on dense path
Watebear Sep 9, 2026
52a9d8e
refactor(minimax-h3): clarify exact ops and add TP8 DMD ref runner
Watebear Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_bf16_4step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
"unload_modules": false,
"attn_type": "flash_attn3",
"rms_type": "torch_native",
"rope_type": "torch_real_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"lora_dynamic_apply": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_bf16_4step_sol.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,12 @@
"strict": true
},
"rms_type": "torch_native",
"rope_type": "torch_real_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"lora_dynamic_apply": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_fp8_4step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_fp8_4step_5090.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_fp8_4step_5090_vae_fp8.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "torch_real_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
Expand All @@ -30,7 +30,7 @@
"vae_attn_type": "sage_attn2",
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_fp8_4step_5090_vae_fp8_sla.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@
"operator": "sage2"
},
"rms_type": "sgl-kernel",
"rope_type": "torch_real_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
"vae_use_compile": true,
"vae_attn_type": "sage_attn2",
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_fp8_4step_5090_vae_fp8_sol.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
"strict": true
},
"rms_type": "sgl-kernel",
"rope_type": "torch_real_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
Expand All @@ -47,7 +47,7 @@
"vae_attn_type": "sage_attn2",
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_fp8_8step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_int8_4step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_int8_convrot_8step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 6.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"dit_quantized": true,
Expand Down
4 changes: 2 additions & 2 deletions configs/minimax_h3/dmd/minimax_h3_ref2av_4step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
"audio_flow_shift": 3.0,
"h3_step_update": "training_euler",
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"text_encoder_tensor_parallel": true,
Expand Down
40 changes: 40 additions & 0 deletions configs/minimax_h3/dmd/minimax_h3_ref2av_turbo_4step_tp8.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"infer_steps": 4,
"target_video_length": 362,
"target_height": 768,
"target_width": 1344,
"fps": 24,
"enable_cfg": false,
"cpu_offload": true,
"reference_image_resize_mode": "match",
"offload_granularity": "model",
"use_adaln_cache": true,
"adaln_cache_dir": "/data/wushuo1/.cache/lightx2v/adaln",
"text_encoder_cpu_offload": true,
"vae_cpu_offload": true,
"lazy_load": false,
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"warmup": false,
"video_flow_shift": 12.0,
"audio_flow_shift": 3.0,
"h3_sampling_profile": "dmd",
"vae_spatial_scale_factor": 16,
"audio_sampling_rate": 32000,
"text_encoder_tensor_parallel": true,
"lora_dynamic_apply": false,
"lora_configs": [
{
"path": "/data/wushuo1/models/Minimax-h3-Turbo/Minimax-h3-Turbo/minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors",
"strength": 1.0,
"alpha": 8
}
],
"parallel": {
"tensor_p_size": 8
}
}
2 changes: 1 addition & 1 deletion configs/minimax_h3/fp8/minimax_h3.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/fp8/minimax_h3_encoder_fp8.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/fp8/minimax_h3_sp_5090.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/fp8/minimax_h3_vae_fp8.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
Expand Down
8 changes: 5 additions & 3 deletions configs/minimax_h3/minimax_h3.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,18 @@
"fps": 24,
"enable_cfg": false,
"cpu_offload": true,
"reference_image_resize_mode": "diffusers",
"offload_granularity": "model",
"use_adaln_cache": true,
"adaln_cache_dir": "~/.cache/lightx2v/adaln",
"text_encoder_cpu_offload": true,
"vae_cpu_offload": true,
"vae_encode_fp32": true,
"lazy_load": false,
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"attn_type": "torch_sdpa",
"rms_type": "torch_native",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/minimax_h3_block_offload.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/minimax_h3_compile.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "torch_real_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": true,
"warmup": true,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/minimax_h3_sol_block_offload.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"strict": true
},
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/minimax_h3_sp.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/minimax_h3_tp.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
2 changes: 1 addition & 1 deletion configs/minimax_h3/minimax_h3_tp_sp.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"unload_modules": false,
"attn_type": "sage_attn2",
"rms_type": "sgl-kernel",
"rope_type": "minimax_h3_triton_rope",
"rope_type": "sgl_exact_neox_rope",
"feature_caching": "NoCaching",
"use_compile": false,
"video_flow_shift": 12.0,
Expand Down
29 changes: 29 additions & 0 deletions lightx2v/common/ops/attn/dynamic_sparse_attn.py
View file
Open in desktop

@Watebear Watebear Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里应该是sparse和packed之后的计算不兼容，所以临时用for循环来替代了。效率肯定低。spare先这样么？

Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,35 @@ def apply(
max_seqlen_kv=None,
**kwargs,
):
if (cu_seqlens_q is None) != (cu_seqlens_kv is None):
raise ValueError("cu_seqlens_q and cu_seqlens_kv must either both be set or both be None")
if q.ndim == 3 and cu_seqlens_q is not None and cu_seqlens_q.numel() > 2:
q_bounds = cu_seqlens_q.tolist()
kv_bounds = cu_seqlens_kv.tolist()
if (
len(q_bounds) != len(kv_bounds)
or q_bounds[0] != 0
or kv_bounds[0] != 0
or q_bounds[-1] != q.shape[0]
or kv_bounds[-1] != k.shape[0]
):
raise ValueError("Invalid packed dynamic sparse attention boundaries")
output = q.new_empty((q.shape[0], q.shape[1] * v.shape[-1]))
for q_start, q_stop, kv_start, kv_stop in zip(q_bounds, q_bounds[1:], kv_bounds, kv_bounds[1:]):
if q_start == q_stop:
continue
segment = self.apply_func(
q[q_start:q_stop],
k[kv_start:kv_stop],
v[kv_start:kv_stop],
None,
None,
q_stop - q_start,
kv_stop - kv_start,
**kwargs,
)
output[q_start:q_stop].copy_(segment)
return output
if max_seqlen_q is None:
max_seqlen_q = q.shape[0]
if max_seqlen_kv is None:
Expand Down
Loading
Loading