Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .ai/references/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ Follow the style introduced in [#14113](https://github.com/huggingface/diffusers
- `torch.nn.MultiheadAttention` is the common instance: it passes `self.out_proj.weight` straight to `torch.nn.functional.multi_head_attention_forward` instead of calling `self.out_proj`, so the hook on `out_proj` never fires. `SiglipVisionModel`'s attention pooling head wraps one — see `tests/pipelines/hunyuan_video/test_hunyuan_video_framepack.py`, whose `image_encoder` is excluded for this reason.
- `HunyuanDiTAttentionPool` (`src/diffusers/models/embeddings.py`) shows the same failure without an MHA module: a plain `nn.Module` that hands its `q_proj` / `k_proj` / `v_proj` / `c_proj` weights to `torch.nn.functional.multi_head_attention_forward`, so all four projections stay offloaded rather than just one. `HunyuanDiT2DModel` opts out of group offloading entirely with `_supports_group_offloading = False`.
- Before adding a skip or an exclusion, confirm the failure still reproduces — several existing skips are stale, having outlived the upstream cause.
- **A migration that surfaces a `src/` gap marks the test `xfail`, it does not patch the pipeline.** Give the marker a module-level name and a `reason` naming the exact gap (`PNDM_*` in `tests/pipelines/pndm/test_pndm.py` is the worked example), and prefer `strict=True` so the marker reports XPASS — and gets deleted — the day the pipeline is fixed. Use `strict=False` only when one mark covers a group whose members do not all fail. Marking a whole test class keeps the mixin's own marks (`@is_memory`, `@require_accelerator`) intact; overriding individual inherited tests drops the decorators they were declared with, so re-declare those too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to xfail things where we need core changes while doing test PRs. xfail is better than skipping the tests.

- **`from_pipe` tests** (a pipeline that is a variant of an existing one — PAG, AnimateDiff, ...) compose the shared `FromPipeTesterMixin` (`tests/pipelines/testing_utils/from_pipe.py`, exported from `..testing_utils`) in their own test class. It derives the original pipeline from `pipeline_class.__name__`; set `original_pipeline_repo` on the test class to pull it from a repo other than the default for that class. The unittest-era `PipelineFromPipeTesterMixin` in `tests/pipelines/test_pipelines_common.py` is what it replaces.
- **A hardware gap is a conditional skip, not an xfail.** When a test fails only because the runner's cuDNN build has no kernel for an op — `RuntimeError: GET was unable to find an engine to execute this computation`, as Sana's depthwise `Conv2d` hits in bfloat16 — wrap the call in `skip_if_no_cudnn_engine()` (`tests/testing_utils.py`). It skips on that error and re-raises every other `RuntimeError`, so the test still runs wherever the kernel exists.
- **PAG pipelines** also compose `PAGPipelineTesterMixin` (`tests/pipelines/pag/testing_utils.py`) in place of `PipelineTesterMixin`: it adds `test_pag_disable_enable` and `test_pag_inference` on top, driven by `base_pipeline_class` and the `pag_*` knobs on the test class. Keep `test_pag_applied_layers` per pipeline — which layers PAG resolves to is model-specific.
- **IP-Adapter tests** live in their own class decorated with `@is_ip_adapter`, subclassing only the config (not `PipelineTesterMixin`). UNet pipelines that load adapters through the standard `IPAdapterMixin` API compose the shared `IPAdapterTesterMixin` (`tests/pipelines/testing_utils/ip_adapter.py`, exported from `..testing_utils`); pipelines whose IP-Adapter API differs (Flux, for example) keep a bespoke mixin next to their own tests.

#### LoRA tests
Expand Down
4 changes: 1 addition & 3 deletions src/diffusers/pipelines/omnigen/pipeline_omnigen.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,7 @@ def retrieve_timesteps(
return timesteps, num_inference_steps


class OmniGenPipeline(
DiffusionPipeline,
):
class OmniGenPipeline(DiffusionPipeline):
r"""
The OmniGen pipeline for multimodal-to-image generation.
Expand Down
39 changes: 4 additions & 35 deletions tests/pipelines/animatediff/test_animatediff.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
AutoencoderKL,
DDIMScheduler,
MotionAdapter,
StableDiffusionPipeline,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just import level changes in the animdatediff series because we moved the from_pipe tests in a dedicated mixin.

UNet2DConditionModel,
)

Expand All @@ -22,16 +21,15 @@
torch_device,
)
from ..pipeline_params import TEXT_TO_IMAGE_BATCH_PARAMS, TEXT_TO_IMAGE_PARAMS
from ..test_pipelines_common import PipelineFromPipeTesterMixin
from ..testing_utils import (
FromPipeTesterMixin,
IPAdapterTesterMixin,
LoraMemoryTesterMixin,
LoraTesterMixin,
MemoryTesterMixin,
UNetLoraTesterMixin,
)
from .testing_utils import (
FROM_PIPE_SKIP_REASON,
FreeInitTesterMixin,
FreeNoiseSplitInferenceTesterMixin,
MotionPipelineTesterConfig,
Expand Down Expand Up @@ -128,32 +126,6 @@ class TestAnimateDiffPipeline(
FreeInitTesterMixin,
FreeNoiseSplitInferenceTesterMixin,
):
def test_from_pipe_consistent_config(self):
original_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"

# create StableDiffusionPipeline
pipe_original = StableDiffusionPipeline.from_pretrained(original_repo, requires_safety_checker=False)

# StableDiffusionPipeline -> AnimateDiffPipeline
pipe_components = self.get_dummy_components()
pipe_additional_components = {
name: component for name, component in pipe_components.items() if name not in pipe_original.components
}
pipe = self.pipeline_class.from_pipe(pipe_original, **pipe_additional_components)

# AnimateDiffPipeline -> StableDiffusionPipeline
original_pipe_additional_components = {}
for name, component in pipe_original.components.items():
if name not in pipe.components or not isinstance(component, pipe.components[name].__class__):
original_pipe_additional_components[name] = component

pipe_original_2 = StableDiffusionPipeline.from_pipe(pipe, **original_pipe_additional_components)

# compare the config
original_config = {k: v for k, v in pipe_original.config.items() if not k.startswith("_")}
original_config_2 = {k: v for k, v in pipe_original_2.config.items() if not k.startswith("_")}
assert original_config_2 == original_config

def test_dict_tuple_outputs_equivalent(self, expected_slice=None, expected_max_difference=1e-4):
if torch_device == "cpu" and expected_slice is None:
# fmt: off
Expand Down Expand Up @@ -244,10 +216,7 @@ def test_animatediff(self):
assert numpy_cosine_similarity_distance(image_slice.flatten(), expected_slice.flatten()) < 1e-3


@pytest.mark.skip(FROM_PIPE_SKIP_REASON)
class TestAnimateDiffPipelineFromPipe(AnimateDiffPipelineTesterConfig, PipelineFromPipeTesterMixin):
"""`from_pipe` forward-pass parity and offload round trip for the AnimateDiff pipeline.
class TestAnimateDiffPipelineFromPipe(AnimateDiffPipelineTesterConfig, FromPipeTesterMixin):
"""`from_pipe` round-trip tests against `StableDiffusionPipeline` for the AnimateDiff pipeline."""

Parked, not deleted: `test_from_pipe_consistent_config` runs for real as a method on the main test class above,
but the forward-pass checks in `PipelineFromPipeTesterMixin` have no pytest-style equivalent yet.
"""
original_pipeline_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"
42 changes: 4 additions & 38 deletions tests/pipelines/animatediff/test_animatediff_controlnet.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import pytest
import torch
from PIL import Image
from transformers import CLIPTextConfig, CLIPTextModel, CLIPTokenizer
Expand All @@ -9,22 +8,20 @@
ControlNetModel,
DDIMScheduler,
MotionAdapter,
StableDiffusionPipeline,
UNet2DConditionModel,
)

from ...testing_utils import torch_device
from ..pipeline_params import TEXT_TO_IMAGE_BATCH_PARAMS, TEXT_TO_IMAGE_PARAMS
from ..test_pipelines_common import PipelineFromPipeTesterMixin
from ..testing_utils import (
FromPipeTesterMixin,
IPAdapterTesterMixin,
LoraMemoryTesterMixin,
LoraTesterMixin,
MemoryTesterMixin,
UNetLoraTesterMixin,
)
from .testing_utils import (
FROM_PIPE_SKIP_REASON,
FreeInitTesterMixin,
FreeNoiseTesterMixin,
MotionPipelineTesterConfig,
Expand Down Expand Up @@ -142,32 +139,6 @@ def get_free_noise_inputs(self):
# `get_dummy_inputs` rather than by overriding `num_frames` on the returned dict.
return self.get_dummy_inputs(num_frames=16)

def test_from_pipe_consistent_config(self):
original_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"

# create StableDiffusionPipeline
pipe_original = StableDiffusionPipeline.from_pretrained(original_repo, requires_safety_checker=False)

# StableDiffusionPipeline -> AnimateDiffControlNetPipeline
pipe_components = self.get_dummy_components()
pipe_additional_components = {
name: component for name, component in pipe_components.items() if name not in pipe_original.components
}
pipe = self.pipeline_class.from_pipe(pipe_original, **pipe_additional_components)

# AnimateDiffControlNetPipeline -> StableDiffusionPipeline
original_pipe_additional_components = {}
for name, component in pipe_original.components.items():
if name not in pipe.components or not isinstance(component, pipe.components[name].__class__):
original_pipe_additional_components[name] = component

pipe_original_2 = StableDiffusionPipeline.from_pipe(pipe, **original_pipe_additional_components)

# compare the config
original_config = {k: v for k, v in pipe_original.config.items() if not k.startswith("_")}
original_config_2 = {k: v for k, v in pipe_original_2.config.items() if not k.startswith("_")}
assert original_config_2 == original_config

def test_dict_tuple_outputs_equivalent(self, expected_slice=None, expected_max_difference=1e-4):
if torch_device == "cpu" and expected_slice is None:
# fmt: off
Expand Down Expand Up @@ -198,12 +169,7 @@ class TestAnimateDiffControlNetPipelineLoRAMemory(AnimateDiffControlNetPipelineT
"""LoRA x memory-optimization tests (group offload, CPU offload) for the pipeline."""


@pytest.mark.skip(FROM_PIPE_SKIP_REASON)
class TestAnimateDiffControlNetPipelineFromPipe(
AnimateDiffControlNetPipelineTesterConfig, PipelineFromPipeTesterMixin
):
"""`from_pipe` forward-pass parity and offload round trip for the AnimateDiff ControlNet pipeline.
class TestAnimateDiffControlNetPipelineFromPipe(AnimateDiffControlNetPipelineTesterConfig, FromPipeTesterMixin):
"""`from_pipe` round-trip tests against `StableDiffusionPipeline` for the AnimateDiff ControlNet pipeline."""

Parked, not deleted: `test_from_pipe_consistent_config` runs for real as a method on the main test class above,
but the forward-pass checks in `PipelineFromPipeTesterMixin` have no pytest-style equivalent yet.
"""
original_pipeline_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"
40 changes: 4 additions & 36 deletions tests/pipelines/animatediff/test_animatediff_sparsectrl.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import pytest
import torch
from PIL import Image
from transformers import CLIPTextConfig, CLIPTextModel, CLIPTokenizer
Expand All @@ -9,22 +8,20 @@
DDIMScheduler,
MotionAdapter,
SparseControlNetModel,
StableDiffusionPipeline,
UNet2DConditionModel,
)

from ...testing_utils import assert_tensors_close, torch_device
from ..pipeline_params import TEXT_TO_IMAGE_BATCH_PARAMS, TEXT_TO_IMAGE_PARAMS
from ..test_pipelines_common import PipelineFromPipeTesterMixin
from ..testing_utils import (
FromPipeTesterMixin,
IPAdapterTesterMixin,
LoraMemoryTesterMixin,
LoraTesterMixin,
MemoryTesterMixin,
UNetLoraTesterMixin,
)
from .testing_utils import (
FROM_PIPE_SKIP_REASON,
FreeInitTesterMixin,
MotionPipelineTesterConfig,
MotionPipelineTesterMixin,
Expand Down Expand Up @@ -138,32 +135,6 @@ class TestAnimateDiffSparseControlNetPipeline(
MotionPipelineTesterMixin,
FreeInitTesterMixin,
):
def test_from_pipe_consistent_config(self):
original_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"

# create StableDiffusionPipeline
pipe_original = StableDiffusionPipeline.from_pretrained(original_repo, requires_safety_checker=False)

# StableDiffusionPipeline -> AnimateDiffSparseControlNetPipeline
pipe_components = self.get_dummy_components()
pipe_additional_components = {
name: component for name, component in pipe_components.items() if name not in pipe_original.components
}
pipe = self.pipeline_class.from_pipe(pipe_original, **pipe_additional_components)

# AnimateDiffSparseControlNetPipeline -> StableDiffusionPipeline
original_pipe_additional_components = {}
for name, component in pipe_original.components.items():
if name not in pipe.components or not isinstance(component, pipe.components[name].__class__):
original_pipe_additional_components[name] = component

pipe_original_2 = StableDiffusionPipeline.from_pipe(pipe, **original_pipe_additional_components)

# compare the config
original_config = {k: v for k, v in pipe_original.config.items() if not k.startswith("_")}
original_config_2 = {k: v for k, v in pipe_original_2.config.items() if not k.startswith("_")}
assert original_config_2 == original_config

def test_dict_tuple_outputs_equivalent(self, expected_slice=None, expected_max_difference=1e-4):
if torch_device == "cpu" and expected_slice is None:
# fmt: off
Expand Down Expand Up @@ -229,12 +200,9 @@ class TestAnimateDiffSparseControlNetPipelineLoRAMemory(
"""LoRA x memory-optimization tests (group offload, CPU offload) for the pipeline."""


@pytest.mark.skip(FROM_PIPE_SKIP_REASON)
class TestAnimateDiffSparseControlNetPipelineFromPipe(
AnimateDiffSparseControlNetPipelineTesterConfig, PipelineFromPipeTesterMixin
AnimateDiffSparseControlNetPipelineTesterConfig, FromPipeTesterMixin
):
"""`from_pipe` forward-pass parity and offload round trip for the AnimateDiff SparseControlNet pipeline.
"""`from_pipe` round-trip tests against `StableDiffusionPipeline` for the AnimateDiff SparseControlNet pipeline."""

Parked, not deleted: `test_from_pipe_consistent_config` runs for real as a method on the main test class above,
but the forward-pass checks in `PipelineFromPipeTesterMixin` have no pytest-style equivalent yet.
"""
original_pipeline_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"
42 changes: 4 additions & 38 deletions tests/pipelines/animatediff/test_animatediff_video2video.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import pytest
import torch
from PIL import Image
from transformers import CLIPTextConfig, CLIPTextModel, CLIPTokenizer
Expand All @@ -8,22 +7,20 @@
AutoencoderKL,
DDIMScheduler,
MotionAdapter,
StableDiffusionPipeline,
UNet2DConditionModel,
)

from ...testing_utils import torch_device
from ..pipeline_params import TEXT_TO_IMAGE_PARAMS, VIDEO_TO_VIDEO_BATCH_PARAMS
from ..test_pipelines_common import PipelineFromPipeTesterMixin
from ..testing_utils import (
FromPipeTesterMixin,
IPAdapterTesterMixin,
LoraMemoryTesterMixin,
LoraTesterMixin,
MemoryTesterMixin,
UNetLoraTesterMixin,
)
from .testing_utils import (
FROM_PIPE_SKIP_REASON,
FreeInitTesterMixin,
FreeNoiseSplitInferenceTesterMixin,
MotionPipelineTesterConfig,
Expand Down Expand Up @@ -133,32 +130,6 @@ def get_free_noise_inputs(self):
inputs["strength"] = 0.5
return inputs

def test_from_pipe_consistent_config(self):
original_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"

# create StableDiffusionPipeline
pipe_original = StableDiffusionPipeline.from_pretrained(original_repo, requires_safety_checker=False)

# StableDiffusionPipeline -> AnimateDiffVideoToVideoPipeline
pipe_components = self.get_dummy_components()
pipe_additional_components = {
name: component for name, component in pipe_components.items() if name not in pipe_original.components
}
pipe = self.pipeline_class.from_pipe(pipe_original, **pipe_additional_components)

# AnimateDiffVideoToVideoPipeline -> StableDiffusionPipeline
original_pipe_additional_components = {}
for name, component in pipe_original.components.items():
if name not in pipe.components or not isinstance(component, pipe.components[name].__class__):
original_pipe_additional_components[name] = component

pipe_original_2 = StableDiffusionPipeline.from_pipe(pipe, **original_pipe_additional_components)

# compare the config
original_config = {k: v for k, v in pipe_original.config.items() if not k.startswith("_")}
original_config_2 = {k: v for k, v in pipe_original_2.config.items() if not k.startswith("_")}
assert original_config_2 == original_config

def test_latent_inputs(self):
pipe = self.get_pipeline().to(torch_device)

Expand Down Expand Up @@ -191,12 +162,7 @@ class TestAnimateDiffVideoToVideoPipelineLoRAMemory(
"""LoRA x memory-optimization tests (group offload, CPU offload) for the pipeline."""


@pytest.mark.skip(FROM_PIPE_SKIP_REASON)
class TestAnimateDiffVideoToVideoPipelineFromPipe(
AnimateDiffVideoToVideoPipelineTesterConfig, PipelineFromPipeTesterMixin
):
"""`from_pipe` forward-pass parity and offload round trip for the AnimateDiff video-to-video pipeline.
class TestAnimateDiffVideoToVideoPipelineFromPipe(AnimateDiffVideoToVideoPipelineTesterConfig, FromPipeTesterMixin):
"""`from_pipe` round-trip tests against `StableDiffusionPipeline` for the AnimateDiff video-to-video pipeline."""

Parked, not deleted: `test_from_pipe_consistent_config` runs for real as a method on the main test class above,
but the forward-pass checks in `PipelineFromPipeTesterMixin` have no pytest-style equivalent yet.
"""
original_pipeline_repo = "hf-internal-testing/tinier-stable-diffusion-pipe"
Loading
Loading