[leejet]

Summary

• Restructure H3 projections and modulation to enable backend fusion, use partial RoPE, and replace audio/video input copies with tensor views.
• Separate prefetch allocation from transfer submission so weight uploads can overlap segment computation.
• Bind segment outputs directly to cached buffers and preserve graph use counts for fusion checks.
• Control backend buffer caching through the runner lifecycle and increase Windows CUDA capacity headroom.
• Update ggml with scheduler callbacks, buffer cache controls, partial RoPE, and CUDA INT8/attention optimizations.

Related Issue / Discussion

N/A

Additional Information

Hardware: NVIDIA GeForce RTX 4090 24 GB, NVIDIA driver 596.36.

DiT files:

Model	DiT file
Mimimax H3	minimax_h3_fl2va_pruned_int8_convrot.safetensors

Sampling throughput in s/it; lower is better:

Model	Resolution	sd.cpp before	sd.cpp after	ComfyUI
Mimimax H3	124f 864 x 480	7.81	3.66	3.83
Mimimax H3	124f 1376 x 768	27.38	16.85	17.77

Configuration:

• Batch 1, CFG 1, Euler, seed 42, steps 20.
• Same DiT files, GGUF text encoders, prompt, and matching sigma schedules across implementations.
• Prompt: a lovely cat holding a sign says 'stable-diffusion.cpp'
• ComfyUI revision cdf272f351, PyTorch 2.13.0+cu130, comfy-kitchen 0.2.40, PyTorch attention.
• sd.cpp before: 77b72a2efed78f55de26892c7e909cee001e8b83.

Checklist

• I have read and confirmed this PR follows the contribution guidelines.

Summary by CodeRabbit

• Performance

  • Improved segmented model execution by reusing cached output buffers and coordinating weight prefetching with computation.
  • Reduced intermediate copying when processing packed audio and video inputs.
  • Improved attention processing efficiency for MiniMax H3 models.
  • Accelerated rotary position processing on supported backends.
  • Added controls for enabling, disabling, and clearing backend buffer caches.

• Reliability

  • Updated device-memory capacity checks to reserve additional headroom, including on Windows with CUDA.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: b7272e67-6447-4f17-8a47-3bbec0984dc1

Commits

Reviewing files that changed from the base of the PR and between a15807d and 301aafa.

Files selected for processing: 1

• ggml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Walkthrough

The runner adds backend buffer-cache controls, graph-cut output binding, revised segmented-execution planning, and staged prefetch submission. MiniMax H3 changes transformer operations and reads video and audio latents as views into packed input. Two local declarations are reformatted without behavior changes.

Changes: Runner memory and prefetch execution

Layer / File(s)	Summary
Backend buffer-cache lifecycle ggml, src/core/ggml_extend_backend.h, src/core/ggml_extend_backend.cpp, src/core/ggml_runner.cpp, examples/server/model_catalog.cpp, src/core/compute_workspace.cpp	The backend API adds functions to enable or clear buffer caching. The runner calls them on its primary and extra backends at startup and shutdown. The ggml subproject reference changes. Two local declarations are reformatted.
Graph-cut output binding and planning src/core/ggml_graph_cut.cpp, src/core/runner_cache.h, src/core/runner_cache.cpp, src/core/ggml_runner.h, src/core/ggml_runner.cpp, src/model_manager.cpp	The runner carries source graph use counts into segment graphs, binds eligible cut outputs to cache storage, and includes those outputs in segment measurement. It updates capacity checks and selects segmented execution based on workspace fit and planning headroom.
Staged prefetch submission src/device_residency_manager.h, src/model_manager.h, src/model_manager_prefetch.cpp, src/core/segment_weight_pipeline.h, src/core/segment_weight_pipeline.cpp, src/core/ggml_runner.cpp	Prefetch preparation and transfer submission are separate operations. The segment pipeline submits queued work through a callback before graph computation. The device-memory margin increases to 1 GiB for CUDA on Windows.

MiniMax H3 model graph

Layer / File(s)	Summary
MLP, attention, and segment processing src/model/diffusion/minimax_h3.hpp	The MLP uses ggml_swiglu and weight-dependent scales. Attention operations are separated into projection, attention, and output projection. Transformer segments are processed separately around shared attention.
Packed audio and video inputs src/model/diffusion/minimax_h3.hpp	The runner validates packed input dimensions and creates video and audio views instead of copying values into separate cached tensors.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant GGMLRunner
  participant SegmentWeightPipeline
  participant DeviceResidencyManager
  participant ModelManager
  SegmentWeightPipeline->>DeviceResidencyManager: prepare_prefetch_params
  DeviceResidencyManager->>ModelManager: prepare prefetched parameters
  GGMLRunner->>SegmentWeightPipeline: submit_next before compute
  SegmentWeightPipeline->>DeviceResidencyManager: submit_prefetched_params
  DeviceResidencyManager->>ModelManager: submit transfers and record event

Loading

Possibly related issues

• [Bug] MiniMax-H3 DiT: MLP fc1 output exceeds Vulkan's 4 GiB buffer limit above 37,449 tokens; the MLP of every layer falls back to CPU #2242: The PR changes the MLP::forward projection and activation path involved in the reported failure. It does not add the issue’s proposed token-axis chunking.

Possibly related PRs

• leejet/stable-diffusion.cpp#2235: Both PRs change GGMLRunner::measure and the segment execution path.
• leejet/stable-diffusion.cpp#2224: Both PRs change graph-cut planning and segmented execution.

Merge Risk: Minimal · up to 301aa

No concrete runtime, integrity, availability, or security regression was established.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title "perf: optimize H3 inference and segmented weight prefetch" matches the main changes. The raw summary shows H3 restructuring (projections, modulation, partial RoPE, tensor views for audio/vi…
Description check		The description follows the template. It has a Summary with five bullets that describe the changes, a Related Issue / Discussion section (marked N/A), an Additional Information section with hardware, …
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title "perf: optimize H3 inference and segmented weight prefetch" matches the main changes. The raw summary shows H3 restructuring (projections, modulation, partial RoPE, tensor views for audio/video latents) and prefetch separation of allocation from transfer submission (prepare_prefetch_params / submit_prefetched_params). It omits the ggml update, graph-cut output binding, and runner buffer-cache control, but it does not describe anything absent from the changeset. The title is concise, specific, and uses a conventional prefix.

Full details: Description check

Explanation

The description follows the template. It has a Summary with five bullets that describe the changes, a Related Issue / Discussion section (marked N/A), an Additional Information section with hardware, DiT file, benchmark tables, and configuration details, and a Checklist with the contribution guidelines item checked. The summary bullets cover the H3 restructuring, prefetch separation, graph-cut output binding, buffer cache control, Windows CUDA headroom, and the ggml update. The benchmark data supports the performance claims.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 36 functions across 16 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}