Feature/qwenimage21 modular

[lucasruan1618]

[Modular] Support Qwen-Image 2.1 generation and multi-reference inpainting

What does this PR do?

Fixes #14832

Adds QwenImage21ModularPipeline using the released Qwen/Qwen-Image-2.1 checkpoint. The workflow is selected from the supplied inputs:

• A prompt runs text-to-image.
• image runs image-conditioned generation and accepts one image or a flat list.
• image and mask_image run single-source inpainting.
• Adding reference_images runs inpainting with additional visual references.

The source is image 1; references follow in the supplied order. Both the text encoder and VAE encode these conditions. Only the source receives a mask. White mask pixels repaint; black pixels restore source latents after each denoising step.

The implementation separates image preparation, text/VAE encoding, denoise preparation, denoising, and decoding into reusable blocks. It uses Diffusers guiders and generation-local KV caches. Named workflows are text2image, image_conditioned, and inpainting.

The existing transformer, VAE, and native pipeline implementation are unchanged. A registry entry for the existing native pipeline enables modular loading from the published checkpoint metadata.

Text2Img

import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image

ASSET_ROOT = "https://raw.githubusercontent.com/lucasruan1618/Image_storage/main/QwenImage21"

pipe = ModularPipeline.from_pretrained("Qwen/Qwen-Image-2.1")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

settings = dict(
    height=512,
    width=512,
    output_resolution=512,
    num_inference_steps=40,
    use_kv_cache=True,
    output="images",
)
images = pipe(
    prompt="A small white ceramic vase on a wooden table against a light grey wall",
    generator=torch.Generator("cpu").manual_seed(123),
    **settings,
)
images[0].save("source.png")

Native pipeline	Modular pipeline

Inpainting with one image

Using the loaded pipeline and settings above:

source = load_image(f"{ASSET_ROOT}/inputs/source.png")
mask = load_image(f"{ASSET_ROOT}/inputs/mask.png").convert("L")

images = pipe(
    prompt=(
        "Replace the white vase with a glossy blue ceramic vase containing three yellow tulips. "
        "Keep the wooden table, wall and lighting unchanged."
    ),
    image=source,
    mask_image=mask,
    strength=1.0,
    generator=torch.Generator("cpu").manual_seed(126),
    **settings,
)
images[0].save("inpaint_single.png")

Source	Mask	Modular single-source inpaint

Multi-image editing

For unmasked editing, pass the source and references together through image. Their list order is also their image
number in the prompt:

source = load_image(f"{ASSET_ROOT}/inputs/source.png")
reference_vase = load_image(f"{ASSET_ROOT}/inputs/reference_vase.png")
reference_flowers = load_image(f"{ASSET_ROOT}/inputs/reference_flowers.png")

images = pipe(
    prompt=(
        "Place the cobalt blue vase from image 2 on the wooden table in image 1, "
        "with the yellow tulips from image 3 inside the vase."
    ),
    image=[source, reference_vase, reference_flowers],
    generator=torch.Generator("cpu").manual_seed(127),
    **settings,
)
images[0].save("three_image_edit_modular.png")

Image 1: source	Image 2: vase	Image 3: flowers	Native pipeline	Modular pipeline

Native and modular decoded outputs match exactly for this three-image edit.

Inpainting with multiple references

images = pipe(
    prompt=(
        "In image 1, replace the white vase with the cobalt blue vase from image 2 "
        "and put the yellow tulips from image 3 into it. "
        "Keep the wooden table, wall, camera and lighting of image 1 unchanged."
    ),
    image=source,
    mask_image=mask,
    reference_images=[
        load_image(f"{ASSET_ROOT}/inputs/reference_vase.png"),
        load_image(f"{ASSET_ROOT}/inputs/reference_flowers.png"),
    ],
    strength=1.0,
    generator=torch.Generator("cpu").manual_seed(126),
    **settings,
)
images[0].save("inpaint_multireference.png")

Source	Mask	Vase reference	Flower reference	Modular multi-reference inpaint

Pretrained test results

• Checkpoint: Qwen/Qwen-Image-2.1, revision b3179ad355be050328e483a9dfdd9e60cd62adfa.
• Hardware/software: one NVIDIA A100-SXM4-80GB, PyTorch 2.8.0+cu128, BF16, eager execution, default attention processor.
• Settings: 512 × 512 RGBA output, condition resolution 512, 40 inference steps, KV cache enabled, CFG disabled, fresh CPU generator for each call.
• Inpainting: strength 1.0, rectangular white mask from (128, 32) through (384, 480); identical source, mask, and seed for the single-source and multi-reference cases. Their prompts differ to describe the references.
• Timing: synchronized end-to-end pipeline calls, including prompt encoding and VAE decoding, after a two-step warmup. Loading and image-file encoding are excluded. Each case runs once; these measurements do not establish a speedup.

Native and modular pipelines share the same loaded model components and use separate equivalent schedulers. Parity compares decoded floating-point arrays before PNG quantization.

Case	Native pipeline	Modular pipeline	Peak allocated memory	Result
Text-to-image, seed 123	3.843 s	3.855 s	31.91 / 31.88 GiB	Exact decoded float output match; max error 0, RMSE 0
Two-image edit using supplied files, seed 129	4.775 s	4.648 s	32.92 / 31.89 GiB	Exact decoded float output and PNG match; max error 0, RMSE 0
Three-image-conditioned generation, seed 127	4.882 s	4.906 s	33.42 / 32.36 GiB	Exact decoded float output match; max error 0, RMSE 0
Single-source inpaint, seed 126	N/A	4.398 s	31.89 GiB	Finite RGBA output; unmasked latent max error 0
Source plus two-reference inpaint, seed 126	N/A	4.977 s	32.37 GiB	Finite RGBA output; unmasked latent max error 0

Self-review

No unresolved blocking implementation issues were found in the completed review against the repository guides. The review checked declared block inputs/outputs, workflow selection and loading, component reuse, guidance, cache lifecycle, tests, and documentation.

Corrections made during review include moving source encoding into its own block, adding named-workflow lookup, rejecting references without an inpainting mask, and aligning the tiny scheduler fixture with the release. All new helpers have active call paths. The temporary pre-normalization text-encoder hook follows the native implementation and is removed in finally; future changes should track that implementation.

Verdict: ready for human review, with the limitations above. AI assisted the implementation and this draft. Human contributor declarations and approval of the exact PR wording remain for the author to complete before posting.

Before submitting

• Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the Coding with AI agents guide?
  • Did you run the self-review skill on the diff?
  • Did you share the final self-review notes in the PR description or a comment?
• Did you read the contributor guideline?
• Was this discussed/approved via a GitHub issue or the forum? [Qwen-Image 2.1] Modular pipeline #14832
• Did you make sure to update the documentation with your changes? (no public API change)
• Did you write any new necessary tests?

Who can review?

• Pipelines and models: @yiyixuxu, @DN6, and @asomoza
• Everyone in the community is also welcome to test and provide feedback.

[github-actions]

Hi @lucasruan1618, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.

[SamuelTallet]

Hello @lucasruan1618, thanks!
You can link #14832 to your PR.

[lucasruan1618]

Hello @lucasruan1618, thanks! You can link #14832 to your PR.

Thanks @SamuelTallet! I've linked the PR to #14832.

[lucasruan1618]

Hi @DN6, @yiyixuxu

I noticed that #14976 has been merged. Thanks for the work on the Qwen-Image 2.1 modular pipeline!

My PR #14820 also includes mask-based inpainting with support for multiple reference images, which isn't covered by #14976.

Would you be interested in adding this feature? I'd be happy to adapt the inpainting implementation on top of the newly merged pipeline and submit it as a separate PR if preferred.

Thanks!