Add EdenLive

[skygoo2000]

What does this PR do?

Adds support for EdenLive, a real-time generative rendering model conditioned on a reference image, an RGB proxy video, and a text prompt. References can come from 2D/3D scenes, existing engines, or recordings.

This integration includes:

• 5B and 14B bidirectional and causal profiles through EdenLivePipeline.
• The EdenLive transformer, DINOv3 proxy conditioning, flow-matching scheduler, and LightTAE decoder.
• Temporal prompts for 5B bidirectional generation, per-chunk prompts for causal generation, and stateful streaming with prompt updates.
• Loading original release checkpoints and saving/reloading standard Diffusers pipelines.
• Documentation, tests, single-GPU and 2/4-GPU offline examples, and a streaming example.

Basic Usage

The example uses a reference image and a proxy video containing at least 49 RGB frames.

import torch
from diffusers import EdenLivePipeline
from diffusers.utils import export_to_video, load_image, load_video

pipe = EdenLivePipeline.from_pretrained(
    "VAST-AI/EdenLive",
    profile="5b-causal",
).to("cuda")

frames = pipe(
    image=load_image("reference.png"),
    video=load_video("proxy-24fps.mp4"),
    prompt="A character explores a snowy landscape.",
    num_frames=49,
    generator=torch.Generator().manual_seed(42),
).frames[0]

export_to_video(frames, "edenlive.mp4", fps=24)

Streaming usage

Initialize a stream with a reference image, then provide 3 sampled RGB proxy images per chunk. This example updates the prompt for the second chunk and exports the resulting 21 frames.

import numpy as np
import torch
from diffusers import EdenLivePipeline
from diffusers.utils import export_to_video, load_image

pipe = EdenLivePipeline.from_pretrained(
    "VAST-AI/EdenLive",
    profile="5b-causal",
).to("cuda")

proxy_paths = [
    ["proxy/chunk-0-0.png", "proxy/chunk-0-1.png", "proxy/chunk-0-2.png"],
    ["proxy/chunk-1-0.png", "proxy/chunk-1-1.png", "proxy/chunk-1-2.png"],
]
prompts = [
    "A character walks through snow.",
    "The character approaches a campfire.",
]

state = pipe.create_stream_state(
    image=load_image("reference.png"),
    prompt=prompts[0],
    generator=torch.Generator().manual_seed(42),
    output_type="np",
)

chunks = []
try:
    for paths, prompt in zip(proxy_paths, prompts):
        result = pipe(
            stream_state=state,
            proxy_frames=[load_image(path) for path in paths],
            prompt=prompt,
        )
        chunks.append(result.frames[0])
finally:
    pipe.close_stream(state)

export_to_video(np.concatenate(chunks, axis=0), "stream.mp4", fps=24)

The Python pipeline uses PyTorch, Transformers with DINOv3 support, and Accelerate. The CLI examples additionally use PyAV and imageio for video I/O.

Demo

comparison.1.mp4

Validation

• 162 EdenLive pipeline tests passed, with 7 passing subtests. These cover model loading, scheduling, decoding, streaming state management, prompt updates, and CPU offloading.
• All 12 multi-GPU tests passed on 2 and 4 A800 GPUs, covering context parallelism with native and Flash Attention, FSDP2, and per-token timesteps.
• Real-weight streaming tests passed for both 5B and 14B causal models. Each model generated 8 consecutive chunks at 128 × 224 resolution, producing 93 frames. Prompts were changed during generation, and cache sizes remained bounded.
• Ruff, dummy object consistency, and copy consistency checks passed.

[github-actions]

Hi @skygoo2000, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.

[skygoo2000]

Hi @yiyixuxu, thanks for adding the label! Could you take a look at this PR, or help suggest the right reviewer for the EdenLive integration? Happy to make any changes needed.

[yiyixuxu]

i left some initial comments
additionally, is it possible to remove the streaming out of the official pipeline?

[yiyixuxu]

can we remove this?
the checkpoint should be converted into diffusers format and then our from_pretrained() should be able to work out of box

[yiyixuxu]

can we remove this?

[yiyixuxu]

can we not override from_pretrained?

[yiyixuxu]

@skygoo2000
the PR deviates a lot from our standard and requirement, are you able to run a self-review? (see guide here https://huggingface.co/docs/diffusers/v0.41.0/en/conceptual/contribution#coding-with-ai-agents)

[skygoo2000]

@skygoo2000 the PR deviates a lot from our standard and requirement, are you able to run a self-review? (see guide here https://huggingface.co/docs/diffusers/v0.41.0/en/conceptual/contribution#coding-with-ai-agents)

Thanks @yiyixuxu!

We've updated the PR to address your comments: removed streaming, additional initialization helper, and the from_pretrained override. Checkpoint conversion is now handled by a standalone script, and both pipelines use standard Diffusers loading.

We also ran the self-review and addressed the integration issues you highlighted. Please take another look when you have a chance!