Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions .ai/references/modular.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,20 +282,22 @@ if block_state.num_frames is None:
block_state.num_frames = 189
```

A declared default is part of the block's contract, so the assembled pipeline is aware of it: the generated docstring shows it and `default_call_parameters` reports it. Resolved inside the body instead, the input renders as `*optional*` with no default, and nothing at the pipeline level can report what the block will actually do. Don't worry about branches of a conditional blockset declaring different defaults for the same input — each branch resolves its own at runtime. Resolve inside `__call__` only when the default is *computed* — derived from other inputs or component config (`height = components.default_sample_size * components.vae_scale_factor`). And when several blocks in a sequence share an input, declare the same default on each (or only on the first block that reads it): in a sequence the input is one shared value, so disagreeing declarations are silently resolved first-block-wins.
A declared default is part of the block's contract, so the assembled pipeline is aware of it: the generated docstring shows it and `default_call_parameters` reports it. Resolved inside the body instead, the input renders as `*optional*` with no default, and nothing at the pipeline level can report what the block will actually do. Don't worry about branches of a conditional blockset declaring different defaults for the same input — each branch resolves its own at runtime. Resolve inside `__call__` only when the default is *computed* — derived from other inputs or component config (`height = components.default_sample_size * components.vae_scale_factor`). And when several blocks in a sequence share an input, declare the same default on each (or only on the first block that reads it): in a sequence the input is one shared value, so the composite warns about disagreeing declarations until it declares the input itself in `input_overrides` (the first declaration wins meanwhile). A conditional block's per-branch defaults count as its opinion in that comparison; a declaration without a default has none.

**A composed `SequentialPipelineBlocks` can override `inputs` / `outputs`.** Two uses: narrow `outputs` to what downstream actually consumes, so the docstring shows the step's product instead of every internal intermediate (Wan-Animate-2's core denoise exposes only `segment_frames`); and change one input default per preset by mapping over `super().inputs` (the distilled core denoise turns `num_inference_steps` into `default=10`):
**A composite replaces what it derives from its sub-blocks with `*_overrides`.** `input_overrides`, `intermediate_output_overrides`, `component_overrides` and `config_overrides` are `{name: declaration}` class attributes on `ConditionalPipelineBlocks`, `SequentialPipelineBlocks` and `LoopSequentialPipelineBlocks` (also `from_blocks_dict` arguments). The name must already be in the derived list, and the declaration, an `InputParam` / `OutputParam` / `ComponentSpec` / `ConfigSpec`, takes the derived entry's place whole, so it states everything: default, `required`, description. Use it to settle a default two sub-blocks disagree on (the composite gets a warning until it does), to set a preset's default, or to reword a description from the composite's point of view. `InputParam.template(...)` keeps a common input to one line.

```python
@property
def inputs(self):
class DistilledCoreDenoiseStep(SequentialPipelineBlocks):
block_classes = [...]
block_names = [...]
# The distilled checkpoint samples in few steps.
return [
InputParam.template("num_inference_steps", default=10) if param.name == "num_inference_steps" else param
for param in super().inputs
]
input_overrides = {"num_inference_steps": InputParam.template("num_inference_steps", default=10)}
```

A composite reads its sub-blocks' declarations and never writes to them: the `required` flag it reports is its own view ("the caller has to pass this"), the leaf's flag is the runtime check in `get_block_state`, and the two legitimately differ. Copy an `InputParam` before changing it.

Overriding the `outputs` property by hand is still the way to narrow `outputs` to what downstream actually consumes, so the docstring shows the step's product instead of every internal intermediate (Wan-Animate-2's core denoise exposes only `segment_frames`).

## ComponentSpec patterns

```python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ There are a few properties you should set:
- `description`: We recommend adding a description for the assembled block to explain what the combined step does.
- `model_name`: This is automatically derived from the sub-blocks but isn't always correct, so you may need to override it.
- `outputs`: By default this is the same as `intermediate_outputs`, but you can manually set it to control which values appear in the doc. This is useful for showing only the final outputs instead of all intermediate values.
- `input_overrides`, `intermediate_output_overrides`, `component_overrides`, `config_overrides`: The composite's own declarations replacing entries of the aggregated lists, as `{name: declaration}`. For example `input_overrides = {"strength": InputParam.template("strength", default=0.9999)}` sets the default the whole sequence uses when two sub-blocks declare different ones. The declaration replaces the aggregated entry whole, and the name must already be in the aggregated list.

These properties, together with the aggregated `inputs`, `intermediate_outputs`, and `expected_components`, are used to automatically generate the `doc` property.

Expand Down
8 changes: 4 additions & 4 deletions src/diffusers/modular_pipelines/anima/modular_blocks_anima.py
Original file line number Diff line number Diff line change
Expand Up @@ -236,15 +236,15 @@ class AnimaAutoCoreDenoiseStep(AutoPipelineBlocks):
The height in pixels of the generated image.
width (`int`, *optional*):
The width in pixels of the generated image.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
sigmas (`list`, *optional*):
Custom sigmas for the denoising process.
strength (`float`, *optional*, defaults to 0.9):
Strength for img2img/inpainting.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
latents (`Tensor`):
latents (`Tensor`, *optional*):
Pre-generated noisy latents for image generation.
**denoiser_input_fields (`None`, *optional*):
The conditional model inputs for the Anima denoiser.
Expand Down Expand Up @@ -342,13 +342,13 @@ class AnimaAutoBlocks(SequentialPipelineBlocks):
The number of images to generate per prompt.
image_latents (`Tensor`, *optional*):
image latents used to guide the image generation. Can be generated from vae_encoder step.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
sigmas (`list`, *optional*):
Custom sigmas for the denoising process.
strength (`float`, *optional*, defaults to 0.9):
Strength for img2img/inpainting.
latents (`Tensor`):
latents (`Tensor`, *optional*):
Pre-generated noisy latents for image generation.
**denoiser_input_fields (`None`, *optional*):
The conditional model inputs for the Anima denoiser.
Expand Down
2 changes: 0 additions & 2 deletions src/diffusers/modular_pipelines/cosmos/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -993,8 +993,6 @@ class Cosmos3TransferDenoiseStep(Cosmos3DenoiseLoopWrapper):
Timestep interval [lo, hi] over which text guidance is active (None = always).
control_guidance_interval (`tuple`, *optional*):
Timestep interval [lo, hi] over which control guidance is active (None = always).
latents (`Tensor`):
Noisy target latents to update.
condition_latents (`Tensor`):
Clean target latents on the conditioned frames (the autoregressive seed).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1068,7 +1068,7 @@ class Cosmos3AutoCoreDenoiseStep(ConditionalPipelineBlocks):
Torch generator for deterministic generation.
fps (`float`, *optional*, defaults to 24.0):
Frame rate of the generated video.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
mixed_precision_format (`str`, *optional*):
None follows the ModelOpt FP8 checkpoint schedule; 'none' keeps the native quantized forward; 'fp8' is
Expand Down Expand Up @@ -1099,7 +1099,7 @@ class Cosmos3AutoCoreDenoiseStep(ConditionalPipelineBlocks):
Latent-frame indexes fixed by visual conditioning.
num_frames (`int`, *optional*):
Number of frames to generate.
latents (`Tensor`):
latents (`Tensor`, *optional*):
Pre-generated noisy vision latents.
sound_latents (`Tensor`, *optional*):
Pre-generated noisy sound latents.
Expand Down Expand Up @@ -1280,7 +1280,7 @@ class Cosmos3OmniBlocks(SequentialPipelineBlocks):
Number of frames the first chunk reuses from the input video.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
mixed_precision_format (`str`, *optional*):
None follows the ModelOpt FP8 checkpoint schedule; 'none' keeps the native quantized forward; 'fp8' is
Expand All @@ -1307,7 +1307,7 @@ class Cosmos3OmniBlocks(SequentialPipelineBlocks):
Vision latents encoded from the conditioning image or video.
vision_condition_frames (`list`, *optional*):
Latent-frame indexes fixed by visual conditioning.
latents (`Tensor`):
latents (`Tensor`, *optional*):
Pre-generated noisy vision latents.
sound_latents (`Tensor`, *optional*):
Pre-generated noisy sound latents.
Expand Down
7 changes: 2 additions & 5 deletions src/diffusers/modular_pipelines/echo/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,8 @@ class EchoDenoiseLoopStep(LoopSequentialPipelineBlocks):
transformer (`LTX2VideoTransformer3DModel`)

Inputs:
sigmas (`list | tuple`):
sigmas (`list | tuple`, *optional*, defaults to (1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725,
0.421875, 0.0)):
DMD sigma schedule, including the terminal zero.
latents (`Tensor`):
Pre-generated noisy latents for image generation.
Expand Down Expand Up @@ -401,10 +402,6 @@ class EchoDenoiseLoopStep(LoopSequentialPipelineBlocks):
Frame rate of the generated video.
attention_kwargs (`dict`, *optional*):
Additional kwargs for attention processors.
audio_latents (`Tensor`):
Packed target audio tokens.
sigmas (`list | tuple`):
DMD sigma schedule including the terminal zero.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
first_frame_tokens (`Tensor`, *optional*):
Expand Down
6 changes: 4 additions & 2 deletions src/diffusers/modular_pipelines/echo/modular_blocks_echo.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@ class EchoDenoiseStep(SequentialPipelineBlocks):
Optional initial audio noise in VAE form (B, C, L, M).
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
sigmas (`list | tuple`):
sigmas (`list | tuple`, *optional*, defaults to (1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725,
0.421875, 0.0)):
DMD sigma schedule, including the terminal zero.
attention_kwargs (`dict`, *optional*):
Additional kwargs for attention processors.
Expand Down Expand Up @@ -263,7 +264,8 @@ class EchoBlocks(SequentialPipelineBlocks):
Optional initial audio noise in VAE form (B, C, L, M).
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
sigmas (`list | tuple`):
sigmas (`list | tuple`, *optional*, defaults to (1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725,
0.421875, 0.0)):
DMD sigma schedule, including the terminal zero.
attention_kwargs (`dict`, *optional*):
Additional kwargs for attention processors.
Expand Down
10 changes: 10 additions & 0 deletions src/diffusers/modular_pipelines/flux/before_denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,16 @@ def intermediate_outputs(self) -> list[OutputParam]:
OutputParam(
"latents", type_hint=torch.Tensor, description="The initial latents to use for the denoising process"
),
OutputParam(
"height",
type_hint=int,
description="The height of the generated image, resolved to the default when not passed",
),
OutputParam(
"width",
type_hint=int,
description="The width of the generated image, resolved to the default when not passed",
),
]

@staticmethod
Expand Down
16 changes: 14 additions & 2 deletions src/diffusers/modular_pipelines/flux/modular_blocks_flux.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,10 @@ class FluxBeforeDenoiseStep(SequentialPipelineBlocks):
Outputs:
latents (`Tensor`):
The initial latents to use for the denoising process
height (`int`):
The height of the generated image, resolved to the default when not passed
width (`int`):
The width of the generated image, resolved to the default when not passed
timesteps (`Tensor`):
The timesteps to use for inference
num_inference_steps (`int`):
Expand Down Expand Up @@ -227,6 +231,10 @@ class FluxImg2ImgBeforeDenoiseStep(SequentialPipelineBlocks):
Outputs:
latents (`Tensor`):
The initial latents to use for the denoising process
height (`int`):
The height of the generated image, resolved to the default when not passed
width (`int`):
The width of the generated image, resolved to the default when not passed
timesteps (`Tensor`):
The timesteps to use for inference
num_inference_steps (`int`):
Expand Down Expand Up @@ -268,9 +276,9 @@ class FluxAutoBeforeDenoiseStep(AutoPipelineBlocks):
scheduler (`FlowMatchEulerDiscreteScheduler`)

Inputs:
height (`int`):
height (`int`, *optional*):
TODO: Add description.
width (`int`):
width (`int`, *optional*):
TODO: Add description.
latents (`Tensor | NoneType`, *optional*):
TODO: Add description.
Expand Down Expand Up @@ -302,6 +310,10 @@ class FluxAutoBeforeDenoiseStep(AutoPipelineBlocks):
Outputs:
latents (`Tensor`):
The initial latents to use for the denoising process
height (`int`):
The height of the generated image, resolved to the default when not passed
width (`int`):
The width of the generated image, resolved to the default when not passed
timesteps (`Tensor`):
The timesteps to use for inference
num_inference_steps (`int`):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,10 @@ class FluxKontextBeforeDenoiseStep(SequentialPipelineBlocks):
Outputs:
latents (`Tensor`):
The initial latents to use for the denoising process
height (`int`):
The height of the generated image, resolved to the default when not passed
width (`int`):
The width of the generated image, resolved to the default when not passed
timesteps (`Tensor`):
The timesteps to use for inference
num_inference_steps (`int`):
Expand Down Expand Up @@ -220,6 +224,10 @@ class FluxKontextImageConditionedBeforeDenoiseStep(SequentialPipelineBlocks):
Outputs:
latents (`Tensor`):
The initial latents to use for the denoising process
height (`int`):
The height of the generated image, resolved to the default when not passed
width (`int`):
The width of the generated image, resolved to the default when not passed
timesteps (`Tensor`):
The timesteps to use for inference
num_inference_steps (`int`):
Expand Down Expand Up @@ -291,6 +299,10 @@ class FluxKontextAutoBeforeDenoiseStep(AutoPipelineBlocks):
Outputs:
latents (`Tensor`):
The initial latents to use for the denoising process
height (`int`):
The height of the generated image, resolved to the default when not passed
width (`int`):
The width of the generated image, resolved to the default when not passed
timesteps (`Tensor`):
The timesteps to use for inference
num_inference_steps (`int`):
Expand Down
6 changes: 3 additions & 3 deletions src/diffusers/modular_pipelines/flux2/modular_blocks_flux2.py
Original file line number Diff line number Diff line change
Expand Up @@ -314,11 +314,11 @@ class Flux2AutoBlocks(SequentialPipelineBlocks):
TODO: Add description.
image_latents (`list`, *optional*):
TODO: Add description.
latents (`Tensor | NoneType`):
latents (`Tensor | NoneType`, *optional*):
TODO: Add description.
num_inference_steps (`None`):
num_inference_steps (`None`, *optional*, defaults to 50):
TODO: Add description.
timesteps (`None`):
timesteps (`None`, *optional*):
TODO: Add description.
sigmas (`None`, *optional*):
TODO: Add description.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -280,13 +280,13 @@ class Flux2KleinAutoCoreDenoiseStep(AutoPipelineBlocks):
TODO: Add description.
width (`int`, *optional*):
TODO: Add description.
latents (`Tensor | NoneType`):
latents (`Tensor | NoneType`, *optional*):
TODO: Add description.
generator (`None`, *optional*):
TODO: Add description.
num_inference_steps (`None`):
num_inference_steps (`None`, *optional*, defaults to 50):
TODO: Add description.
timesteps (`None`):
timesteps (`None`, *optional*):
TODO: Add description.
sigmas (`None`, *optional*):
TODO: Add description.
Expand Down Expand Up @@ -355,11 +355,11 @@ class Flux2KleinAutoBlocks(SequentialPipelineBlocks):
TODO: Add description.
image_latents (`list`, *optional*):
TODO: Add description.
latents (`Tensor | NoneType`):
latents (`Tensor | NoneType`, *optional*):
TODO: Add description.
num_inference_steps (`None`):
num_inference_steps (`None`, *optional*, defaults to 50):
TODO: Add description.
timesteps (`None`):
timesteps (`None`, *optional*):
TODO: Add description.
sigmas (`None`, *optional*):
TODO: Add description.
Expand Down
Loading
Loading