Skip to content

[docs] Refactor get started section #14783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stevhliu merged 3 commits into from
Sep 17, 2026
Merged

[docs] Refactor get started section #14783

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
80b5c3a
docs
stevhliu Sep 15, 2026
e72850c
feedback
stevhliu Sep 16, 2026
c8342bf
feedback
stevhliu Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 18 additions & 8 deletions docs/source/en/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,28 @@ specific language governing permissions and limitations under the License.

# Diffusers

Diffusers is a library of state-of-the-art pretrained diffusion models for generating videos, images, and audio.
Diffusers provides pretrained diffusion models and the building blocks for custom image, video, and audio workflows.

The library revolves around the [`DiffusionPipeline`], an API designed for:
It has two main paths.

- easy inference with only a few lines of code
- flexibility to mix-and-match pipeline components (models, schedulers)
- loading and using adapters like LoRA
- [`DiffusionPipeline`] supports few-line inference with pretrained checkpoints, plus adapters like LoRA. This is the easy path for generation.
- [Modular Diffusers](./modular_diffusers/overview) enables composable blocks and [`ModularPipeline`] for custom pipelines when you need more control.

Diffusers also comes with optimizations - such as offloading and quantization - to ensure even the largest models are accessible on memory-constrained devices. If memory is not an issue, Diffusers supports torch.compile to boost inference speed.
Optimizations such as offloading and quantization keep large models runnable on memory-constrained devices. If memory is not an issue, Diffusers also supports `torch.compile` for faster inference.

Get started right away with a Diffusers model on the [Hub](https://huggingface.co/models?library=diffusers&sort=trending) today!
Browse trending Diffusers models on the [Hub](https://huggingface.co/models?library=diffusers&sort=trending) now.

## Learn

If you're a beginner, we recommend starting with the [Hugging Face Diffusion Models Course](https://huggingface.co/learn/diffusion-course/unit0/1). You'll learn the theory behind diffusion models, and learn how to use the Diffusers library to generate images, fine-tune your own models, and more.
If you're a beginner, start with the [Hugging Face Diffusion Models Course](https://huggingface.co/learn/diffusion-course/unit0/1). It covers diffusion theory and how to generate images, fine-tune models, and more with Diffusers.

The [Quickstart](./quicktour) also includes a copyable agent setup prompt for inference.

## Where next
Comment thread
stevhliu marked this conversation as resolved.

- [Inference](./using-diffusers/loading) — load pipelines and run generation
- [Optimize and scale](./stable_diffusion) — memory, speed, quantization, and serving
- [Modular Diffusers](./modular_diffusers/overview) — build custom pipelines from blocks
- [Train and fine-tune](./training/overview) — train diffusion models and adapters
- [CLI](./using-diffusers/cli) - run and package pipelines from the command line

6 changes: 3 additions & 3 deletions docs/source/en/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ specific language governing permissions and limitations under the License.

# Installation

Diffusers is tested on Python 3.8+ and PyTorch 2.6+. Install [PyTorch](https://pytorch.org/get-started/locally/) according to your system and setup.
Diffusers requires Python 3.10+ and is tested with PyTorch 2.6+. Install [PyTorch](https://pytorch.org/get-started/locally/) according to your system and setup.

Create a [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/) for easier management of separate projects and to avoid compatibility issues between dependencies. Use [uv](https://docs.astral.sh/uv/), a Rust-based Python package and project manager, to create a virtual environment and install Diffusers.

Expand All @@ -26,8 +26,6 @@ Install Diffusers with one of the following methods.
<hfoptions id="install">
<hfoption id="pip">

PyTorch only supports Python 3.8 - 3.11 on Windows.

```bash
uv pip install diffusers["torch"] transformers
```
Expand Down Expand Up @@ -77,6 +75,8 @@ uv pip install git+https://github.com/huggingface/diffusers
</hfoption>
</hfoptions>

Install with extras when you need them. `diffusers["torch"]` pulls PyTorch and Accelerate. Other extras cover quantization backends, training, and docs. See [`setup.py`](https://github.com/huggingface/diffusers/blob/main/setup.py) for the full list.

## Devices

Diffusers runs on any accelerator supported by PyTorch. The examples throughout the docs use `"cuda"` because it is the most common setup, but nothing is CUDA-specific. Swap in the device string for your hardware, such as `"xpu"` for Intel GPUs, `"mps"` for Apple silicon, or `"cpu"`.
Expand Down
119 changes: 33 additions & 86 deletions docs/source/en/quicktour.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,34 +14,36 @@ specific language governing permissions and limitations under the License.

Diffusers is a library for developers and researchers that provides an easy inference API for generating images, videos and audio, as well as the building blocks for implementing new workflows.

Diffusers provides many optimizations out-of-the-box that makes it possible to load and run large models on setups with limited memory or to accelerate inference.
Diffusers provides many optimizations out-of-the-box that make it possible to load and run large models on setups with limited memory or to accelerate inference.

This Quickstart will give you an overview of Diffusers and get you up and generating quickly.

> [!TIP]
> Before you begin, make sure you have a Hugging Face [account](https://huggingface.co/join) in order to use gated models like [Flux](https://huggingface.co/black-forest-labs/FLUX.1-dev).
> Before you begin, make sure you have a Hugging Face [account](https://huggingface.co/join) to use gated models like [Flux](https://huggingface.co/black-forest-labs/FLUX.1-dev). Building a custom pipeline instead? See the [Modular Diffusers quickstart](./modular_diffusers/quickstart) and [overview](./modular_diffusers/overview).

Follow the [Installation](./installation) guide to install Diffusers if it's not already installed.

## DiffusionPipeline
## Agent prompt

A diffusion model combines multiple components to generate outputs in any modality based on an input, such as a text description, image or both.
Paste this into your coding agent to get Diffusers set up for inference.

For a standard text-to-image model:
```text
Help me get set up with Hugging Face Diffusers for inference.

1. A text encoder turns a prompt into embeddings that guide the denoising process. Some models have more than one text encoder.
2. A scheduler contains the algorithmic specifics for gradually denoising initial random noise into clean outputs. Different schedulers affect generation speed and quality.
3. A UNet or diffusion transformer (DiT) is the workhorse of a diffusion model.
1. Install Diffusers for my environment with `uv pip install "diffusers[torch]"`.
2. If I need gated Hub models, help me authenticate to the Hugging Face Hub.
3. Install Diffusers coding agent skills with `diffusers-cli skills add diffusers-cli`. Pass `--cursor`, `--claude`, or `--codex` if auto-detect fails. Optionally pass `--all` to install every skill in the registry.
4. Run a first text-to-image with DiffusionPipeline or `diffusers-cli run`, using a small or current Quickstart model and the right `device_map` for my machine.
5. Ask what I want next and point me at the matching docs.
```

At each step, it performs the denoising predictions, such as how much noise to remove or the general direction in which to steer the noise to generate better quality outputs.
## DiffusionPipeline

The UNet or DiT repeats this loop for a set amount of steps to generate the final output.

4. A variational autoencoder (VAE) encodes and decodes pixels to a spatially compressed latent-space. *Latents* are compressed representations of an image and are more efficient to work with. The UNet or DiT operates on latents, and the clean latents at the end are decoded back into images.
[`DiffusionPipeline`] packages the pieces of a diffusion model (text encoder, scheduler, UNet or DiT, and VAE) into one class for inference. Load with [`~DiffusionPipeline.from_pretrained`], then call the pipeline.

The [`DiffusionPipeline`] packages all these components into a single class for inference. There are several arguments in [`~DiffusionPipeline.__call__`] you can change, such as `num_inference_steps`, that affect the diffusion process. Try different values and arguments to see how they change generation quality or speed.
Arguments on [`~DiffusionPipeline.__call__`] such as `num_inference_steps` change quality and speed. For loading details and mix-and-match components, see [Load pipelines](./using-diffusers/loading). To swap the scheduler, see [Schedulers](./using-diffusers/schedulers).

Load a model with [`~DiffusionPipeline.from_pretrained`] and describe what you'd like to generate. The example below uses the default argument values.
The examples below use the default argument values.

<hfoptions id="diffusionpipeline">
<hfoption id="text-to-image">
Expand Down Expand Up @@ -71,7 +73,6 @@ Use `.frames[0]` to access the generated video output and [`~utils.export_to_vid
```py
import torch
from diffusers import AutoencoderKLWan, DiffusionPipeline
from diffusers.quantizers import PipelineQuantizationConfig
from diffusers.utils import export_to_video

vae = AutoencoderKLWan.from_pretrained(
Expand All @@ -81,7 +82,7 @@ vae = AutoencoderKLWan.from_pretrained(
)
pipeline = DiffusionPipeline.from_pretrained(
"Wan-AI/Wan2.2-T2V-A14B-Diffusers",
vae=vae
vae=vae,
dtype=torch.bfloat16,
device_map="cuda" # or "mps", "xpu", "cpu"
)
Expand All @@ -101,9 +102,7 @@ export_to_video(video, "output.mp4", fps=16)

## LoRA

Adapters insert a small number of trainable parameters to the original base model. Only the inserted parameters are fine-tuned while the rest of the model weights remain frozen. This makes it fast and cheap to fine-tune a model on a new style. Among adapters, [LoRAs](./tutorials/using_peft_for_inference) are the most popular.

Add a LoRA to a pipeline with the [`~loaders.QwenImageLoraLoaderMixin.load_lora_weights`] method. Some LoRAs require a special word to trigger them, such as `Realism`, in the example below. Check a LoRA's model card to see if it requires a trigger word.
[LoRA](./tutorials/using_peft_for_inference) adapters add a small style or subject checkpoint on top of a base pipeline. Load one with [`~loaders.QwenImageLoraLoaderMixin.load_lora_weights`]. Some LoRAs need a trigger phrase. Check the LoRA's model card.

```py
import torch
Expand All @@ -113,25 +112,21 @@ pipeline = DiffusionPipeline.from_pretrained(
"Qwen/Qwen-Image", dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu"
)
pipeline.load_lora_weights(
"flymy-ai/qwen-image-realism-lora",
"threecrowco/VolkClipartQwen",
weight_name="pytorch_lora_weights.safetensors",
)

prompt = """
super Realism cinematic film still of a cat sipping a margarita in a pool in Palm Springs in the style of umempart, California
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
Volk clipart style drawing of a cat sipping a margarita in a pool in Palm Springs, California, flat colors, bold outlines, simple shapes
"""
pipeline(prompt).images[0]
```

Check out the [LoRA](./tutorials/using_peft_for_inference) docs or Adapters section to learn more.
## Quantization and optimizations

## Quantization
Large models often need less memory or more speed. Use [quantization](./quantization/overview) to shrink weights in memory, and [`~ModelMixin.compile_repeated_blocks`] to speed up later generates. For [model offloading](./optimization/memory#model-offloading) and other options, see [Optimize and scale](./stable_diffusion).

[Quantization](./quantization/overview) stores data in fewer bits to reduce memory usage. It may also speed up inference because it takes less time to perform calculations with fewer bits.

Diffusers provides several quantization backends and picking one depends on your use case. For example, [bitsandbytes](./quantization/bitsandbytes) and [torchao](./quantization/torchao) are both simple and easy to use for inference, but torchao supports more [quantization types](./quantization/torchao#supported-quantization-types) like fp8.

Configure [`PipelineQuantizationConfig`] with the backend to use, the specific arguments (refer to the [API](./api/quantization) reference for available arguments) for that backend, and which components to quantize. The example below quantizes the model to 4-bits and only uses 14.93GB of memory.
To use less memory, load in 4-bit with bitsandbytes.

```py
import torch
Expand All @@ -155,61 +150,9 @@ cinematic film still of a cat sipping a margarita in a pool in Palm Springs, Cal
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
"""
pipeline(prompt).images[0]
print(f"Max memory reserved: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB")
```

Take a look at the [Quantization](./quantization/overview) section for more details.

## Optimizations

> [!TIP]
> Optimization is dependent on hardware specs such as memory. Use this [Space](https://huggingface.co/spaces/diffusers/optimized-diffusers-code) to generate code examples that include all of Diffusers' available memory and speed optimization techniques for any model you're using.

Modern diffusion models are very large and have billions of parameters. The iterative denoising process is also computationally intensive and slow. Diffusers provides techniques for reducing memory usage and boosting inference speed. These techniques can be combined with quantization to optimize for both memory usage and inference speed.

### Memory usage

The text encoders and UNet or DiT can use up as much as ~30GB of memory, exceeding the amount available on many free-tier or consumer GPUs.

Offloading stores weights that aren't currently used on the CPU and only moves them to the GPU when they're needed. There are a few offloading types and the example below uses [model offloading](./optimization/memory#model-offloading). This moves an entire model, like a text encoder or transformer, to the CPU when it isn't actively being used.

Call [`~DiffusionPipeline.enable_model_cpu_offload`] to activate it. By combining quantization and offloading, the following example only requires ~12.54GB of memory.

```py
import torch
from diffusers import DiffusionPipeline
from diffusers.quantizers import PipelineQuantizationConfig

quant_config = PipelineQuantizationConfig(
quant_backend="bitsandbytes_4bit",
quant_kwargs={"load_in_4bit": True, "bnb_4bit_quant_type": "nf4", "bnb_4bit_compute_dtype": torch.bfloat16},
components_to_quantize=["transformer", "text_encoder"],
)
pipeline = DiffusionPipeline.from_pretrained(
"Qwen/Qwen-Image",
dtype=torch.bfloat16,
quantization_config=quant_config,
device_map="cuda" # or "mps", "xpu", "cpu"
)
pipeline.enable_model_cpu_offload()

prompt = """
cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
"""
pipeline(prompt).images[0]
print(f"Max memory reserved: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB")
```

Refer to the [Reduce memory usage](./optimization/memory) docs to learn more about other memory reducing techniques.

### Inference speed

The denoising loop performs a lot of computations and can be slow. Methods like [torch.compile](./optimization/fp16#torchcompile) increases inference speed by compiling the computations into an optimized kernel. Compilation is slow for the first generation but successive generations should be much faster.

The example below uses [regional compilation](./optimization/fp16#regional-compilation) to only compile small regions of a model. It reduces cold-start latency while also providing a runtime speed up.

Call [`~ModelMixin.compile_repeated_blocks`] on the model to activate it.
To speed up later runs, compile repeated blocks on the transformer. The first generate after compile is a cold start and is slow. Later generates are faster.

```py
import torch
Expand All @@ -218,15 +161,19 @@ from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained(
"Qwen/Qwen-Image", dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu"
)
pipeline.transformer.compile_repeated_blocks(fullgraph=True)

pipeline.transformer.compile_repeated_blocks(
fullgraph=True,
)
prompt = """
cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
"""
pipeline(prompt).images[0]
```

Check out the [Accelerate inference](./optimization/fp16) or [Caching](./optimization/cache) docs for more methods that speed up inference.
## Next steps

- [Inference](./using-diffusers/loading) — pipelines, prompting, and adapters
- [Optimize and scale](./stable_diffusion) — memory, speed, quantization, and serving
- [Modular Diffusers](./modular_diffusers/overview) — composable blocks and custom pipelines
- [Train and fine-tune](./training/overview) — training scripts and adapters
- [CLI](./using-diffusers/cli) — generate from the command line
Loading
Loading