Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ API and command-line option may change frequently.***

## 🔥Important News

* **2026/08/20** 🚀 stable-diffusion.cpp now supports **LTX-2.5**
* **2026/08/04** 🚀 stable-diffusion.cpp adds **Day-1 support for MiniMax-H3**
* **2026/06/25** 🚀 stable-diffusion.cpp now supports **Krea2**
* **2026/06/04** 🚀 stable-diffusion.cpp now supports **Ideogram4**
Expand Down Expand Up @@ -68,7 +69,7 @@ API and command-line option may change frequently.***
- Video Models
- [Wan2.1/Wan2.2](./docs/wan.md)
- [MiniMax-H3](./docs/minimax_h3.md)
- [LTX-2.3](./docs/ltx2.md)
- [LTX-2.3/LTX-2.5](./docs/ltx2.md)
- [HunyuanVideo 1.5](./docs/hunyuan_video.md)
- [LingBot-Video](./docs/lingbot_video.md)
- [PhotoMaker](./docs/photo_maker.md) support.
Expand Down
88 changes: 87 additions & 1 deletion docs/ltx2.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,91 @@
# How to Use

Both LTX-2.3 and LTX-2.5 are supported. The two share a transformer, video VAE and audio
VAE architecture; LTX-2.5 drops the video FFN biases, adds a learned keyframe
absolute-position embedding, and pairs with a Gemma 4 text encoder instead of Gemma 3.
Everything is detected from the weights, so the command lines differ only in which files
you pass.

# LTX-2.5

## Download weights

Weights live in the [LTX-2.5 repository](https://huggingface.co/Lightricks/LTX-2.5), which
publishes one file per component.

- Download the transformer (`dev` for the guided pipelines, `distilled` for few-step runs)
- safetensors: https://huggingface.co/Lightricks/LTX-2.5/tree/main/diffusion_models
- gguf: https://huggingface.co/vantagewithai/LTX-2.5-GGUF/tree/main
- Download the text encoder. This is a Gemma 4 12B fine-tuned for LTX with the text
projection bundled in, so no separate `--embeddings-connectors` file is needed. Google's
stock Gemma 4 is not a substitute.
- safetensors: https://huggingface.co/Lightricks/LTX-2.5/blob/main/text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors
- Download the video vae. Use the **conv** variant: `ltx-2.5-video-vae-conv-bf16.safetensors`.
The default `ltx-2.5-video-vae-bf16.safetensors` is a diffusion decoder, which is not
implemented here.
- safetensors: https://huggingface.co/Lightricks/LTX-2.5/blob/main/vae/ltx-2.5-video-vae-conv-bf16.safetensors
- Download the audio vae
- safetensors: https://huggingface.co/Lightricks/LTX-2.5/blob/main/vae/ltx-2.5-audio-vae-bf16.safetensors
- Download the LTX spatial latent upscaler
- safetensors: https://huggingface.co/Lightricks/LTX-2.5/blob/main/latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors
- Download the LTX temporal latent upscaler for standalone frame interpolation
- safetensors: https://huggingface.co/Lightricks/LTX-2.5/blob/main/latent_upscale_models/ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors

To run the text encoder quantized, convert it once with sd-cli:

```
.\bin\Release\sd-cli.exe -M convert -m ..\models\text_encoders\gemma4-12b-with-proj-ltx-2.5-bf16.safetensors --type q8_0 -o ..\models\text_encoders\gemma4-12b-with-proj-ltx-2.5-Q8_0.gguf
```

## Examples

### LTX-2.5 dev T2V

```
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\ltx-2.5-22b-dev-transformer-Q8_0.gguf --vae ..\models\vae\ltx-2.5-video-vae-conv-bf16.safetensors --audio-vae ..\models\vae\ltx-2.5-audio-vae-bf16.safetensors --llm ..\models\text_encoders\gemma4-12b-with-proj-ltx-2.5-Q8_0.gguf -p "A wide aerial shot of a red vintage convertible driving along a coastal cliff road at sunset, waves crashing below" --cfg-scale 3.0 --sampling-method euler -v -n "worst quality, low quality, blurry, distorted, artifacts" -W 1280 -H 720 --diffusion-fa --offload-to-cpu --video-frames 121 --fps 24 -o t2v.webm
```

### LTX-2.5 dev I2V

```
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\ltx-2.5-22b-dev-transformer-Q8_0.gguf --vae ..\models\vae\ltx-2.5-video-vae-conv-bf16.safetensors --audio-vae ..\models\vae\ltx-2.5-audio-vae-bf16.safetensors --llm ..\models\text_encoders\gemma4-12b-with-proj-ltx-2.5-Q8_0.gguf -p "a lovely cat blinking slowly, gentle camera push in" --cfg-scale 3.0 --sampling-method euler -v -W 1280 -H 720 --diffusion-fa --offload-to-cpu --video-frames 121 -i ..\assets\ernie_image\turbo_example.png -o i2v.webm
```

### LTX-2.5 spatial latent upscale

Works exactly like the LTX-2.3 upscaler described below; put
`ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors` under `--hires-upscalers-dir` and
pass its name without path or extension to `--hires-upscaler`.

```
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\ltx-2.5-22b-dev-transformer-Q8_0.gguf --vae ..\models\vae\ltx-2.5-video-vae-conv-bf16.safetensors --audio-vae ..\models\vae\ltx-2.5-audio-vae-bf16.safetensors --llm ..\models\text_encoders\gemma4-12b-with-proj-ltx-2.5-Q8_0.gguf --hires-upscalers-dir ..\models\latent_upscale_models --hires-upscaler ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0 --hires --hires-steps 6 -p "a lovely cat" --cfg-scale 3.0 --sampling-method euler -v -W 640 -H 360 --diffusion-fa --offload-to-cpu --video-frames 121 -o hires_t2v.webm
```

### Standalone video enhancement

`ltx_upscale` runs only the Conv VAE and LTX latent upscalers: it does not load or
run the LTX transformer, text encoder, prompt conditioning, or sampler. Input video is
an image-frame directory because sd-cli does not include a general video decoder. Frames
must be equally sized and lexically ordered (for example `0000.png`, `0001.png`, ...).
LTX requires dimensions divisible by 32. If the source size is not compatible, pass
`-W` and `-H` to resize all frames during loading. The command pads the final input
frame internally for the VAE, then trims output back to the source duration (or `2T-1`
frames with temporal interpolation).

```
.\bin\Release\sd-cli.exe -M ltx_upscale --vae ..\models\vae\ltx-2.5-video-vae-conv-bf16.safetensors --input-video ..\frames --ltx-spatial-upscaler ..\models\latent_upscale_models\ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors --ltx-temporal-upscaler ..\models\latent_upscale_models\ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors --fps 24 -o enhanced.webm
```

Either upscaler option may be omitted. Spatial upscaling doubles width and height;
temporal upscaling produces `2T-1` video frames, so double `--fps` for frame interpolation.

## Not implemented

- The diffusion video decoder (`ltx-2.5-video-vae-bf16.safetensors`). Use the conv VAE.
- The duration head (`--auto-duration`); pass `--video-frames` explicitly for generation.

# LTX-2.3

## Download weights

- Download LTX-2.3
Expand Down Expand Up @@ -74,4 +160,4 @@ By default, the hires refine pass uses the main sampler and scheduler, then trim
src="../assets/ltx2/hires_i2v.webm"
controls
muted
style="max-width: 100%; height: auto;"></video>
style="max-width: 100%; height: auto;"></video>
37 changes: 36 additions & 1 deletion examples/cli/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -524,7 +524,7 @@ bool save_results(const SDCliParams& cli_params,
return sucessful_reults != 0;
}

if (cli_params.mode == VID_GEN && num_results > 1) {
if ((cli_params.mode == VID_GEN || cli_params.mode == LTX_UPSCALE) && num_results > 1) {
if (ext_lower != ".avi" && ext_lower != ".webp" && ext_lower != ".webm")
ext = ".avi";
fs::path video_path = base_path;
Expand Down Expand Up @@ -882,6 +882,22 @@ int main(int argc, const char* argv[]) {
}
}

if (cli_params.mode == LTX_UPSCALE) {
gen_params.control_frames.clear();
if (!load_images_from_dir(gen_params.input_video_path,
gen_params.control_frames,
gen_params.width_and_height_are_set() ? gen_params.width : 0,
gen_params.width_and_height_are_set() ? gen_params.height : 0,
0,
cli_params.verbose) || gen_params.control_frames.empty()) {
LOG_ERROR("load LTX video upscale input frames from '%s' failed", gen_params.input_video_path.c_str());
return 1;
}
gen_params.video_frames = static_cast<int>(gen_params.control_frames.size());
gen_params.set_width_and_height_if_unset(gen_params.control_frames[0].get().width,
gen_params.control_frames[0].get().height);
}

if (!gen_params.pm_id_images_dir.empty()) {
gen_params.pm_id_images.clear();
if (!load_images_from_dir(gen_params.pm_id_images_dir,
Expand Down Expand Up @@ -946,6 +962,25 @@ int main(int argc, const char* argv[]) {
generated_video = nullptr;
}
results.adopt(generated_video, num_results);
} else if (cli_params.mode == LTX_UPSCALE) {
sd_ltx_upscale_video_params_t params;
sd_ltx_upscale_video_params_init(&params);
gen_params.control_frame_views.clear();
gen_params.control_frame_views.reserve(gen_params.control_frames.size());
for (auto& frame : gen_params.control_frames) {
gen_params.control_frame_views.push_back(frame.get());
}
params.input_frames = gen_params.control_frame_views.data();
params.input_frame_count = static_cast<int>(gen_params.control_frame_views.size());
params.spatial_upscaler_path = gen_params.ltx_spatial_upscaler_path.empty() ? nullptr : gen_params.ltx_spatial_upscaler_path.c_str();
params.temporal_upscaler_path = gen_params.ltx_temporal_upscaler_path.empty() ? nullptr : gen_params.ltx_temporal_upscaler_path.c_str();
params.vae_tiling_params = gen_params.vae_tiling_params;
sd_image_t* generated_video = nullptr;
if (!ltx_upscale_video(sd_ctx.get(), &params, &generated_video, &num_results)) {
generated_video = nullptr;
num_results = 0;
}
results.adopt(generated_video, num_results);
}

if (!results) {
Expand Down
33 changes: 30 additions & 3 deletions examples/common/common.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ const char* const modes_str[] = {
"img_gen",
"adetailer",
"vid_gen",
"ltx_upscale",
"convert",
"upscale",
"metadata",
Expand Down Expand Up @@ -738,7 +739,7 @@ bool SDContextParams::validate(SDMode mode) {
LOG_ERROR("error: convert mode needs at least one model input path\n");
return false;
}
} else if (mode != UPSCALE && mode != METADATA && model_path.length() == 0 && diffusion_model_path.length() == 0) {
} else if (mode != UPSCALE && mode != METADATA && mode != LTX_UPSCALE && model_path.length() == 0 && diffusion_model_path.length() == 0 && vae_path.length() == 0) {
LOG_ERROR("error: the following arguments are required: model_path/diffusion_model\n");
return false;
}
Expand Down Expand Up @@ -984,6 +985,21 @@ ArgOptions SDGenerationParams::get_options() {
"such as 00.png, 01.png, ... etc.",
0,
&control_video_path},
{"",
"--input-video",
"path to input video frames for ltx_upscale. It must be a directory of equally sized images in lexicographical order.",
0,
&input_video_path},
{"",
"--ltx-spatial-upscaler",
"path to an LTX latent spatial upscaler model for ltx_upscale",
0,
&ltx_spatial_upscaler_path},
{"",
"--ltx-temporal-upscaler",
"path to an LTX latent temporal upscaler model for ltx_upscale",
0,
&ltx_temporal_upscaler_path},
{"",
"--pm-id-images-dir",
"path to PHOTOMAKER input id images dir",
Expand Down Expand Up @@ -2405,11 +2421,11 @@ bool SDGenerationParams::validate(SDMode mode) {
}
}

if (mode == VID_GEN && video_frames <= 0) {
if ((mode == VID_GEN || mode == LTX_UPSCALE) && video_frames <= 0) {
return false;
}

if (mode == VID_GEN && fps <= 0) {
if ((mode == VID_GEN || mode == LTX_UPSCALE) && fps <= 0) {
return false;
}

Expand All @@ -2423,6 +2439,17 @@ bool SDGenerationParams::validate(SDMode mode) {
return false;
}

if (mode == LTX_UPSCALE) {
if (input_video_path.empty()) {
LOG_ERROR("error: ltx_upscale mode requires --input-video");
return false;
}
if (ltx_spatial_upscaler_path.empty() && ltx_temporal_upscaler_path.empty()) {
LOG_ERROR("error: ltx_upscale mode requires --ltx-spatial-upscaler and/or --ltx-temporal-upscaler");
return false;
}
}

if (sample_params.shifted_timestep < 0 || sample_params.shifted_timestep > 1000) {
LOG_ERROR("error: shifted_timestep must be in range [0, 1000]");
return false;
Expand Down
6 changes: 5 additions & 1 deletion examples/common/common.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@
#define BOOL_STR(b) ((b) ? "true" : "false")

extern const char* const modes_str[];
#define SD_ALL_MODES_STR "img_gen, adetailer, vid_gen, convert, upscale, metadata"
#define SD_ALL_MODES_STR "img_gen, adetailer, vid_gen, ltx_upscale, convert, upscale, metadata"

enum SDMode {
IMG_GEN,
ADETAILER,
VID_GEN,
LTX_UPSCALE,
CONVERT,
UPSCALE,
METADATA,
Expand Down Expand Up @@ -216,6 +217,9 @@ struct SDGenerationParams {
std::vector<std::string> ref_video_audio_paths;
std::vector<std::string> ref_audio_paths;
std::string control_video_path;
std::string input_video_path;
std::string ltx_spatial_upscaler_path;
std::string ltx_temporal_upscaler_path;

sd_sample_params_t sample_params;
sd_sample_params_t high_noise_sample_params;
Expand Down
15 changes: 15 additions & 0 deletions include/stable-diffusion.h
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,16 @@ typedef struct {
bool circular_y;
} sd_vid_gen_params_t;

// Standalone LTX video enhancement. The input is a decoded sequence of RGB frames;
// the caller owns input_frames and must release output_frames with free_sd_images().
typedef struct {
const sd_image_t* input_frames;
int input_frame_count;
const char* spatial_upscaler_path;
const char* temporal_upscaler_path;
sd_tiling_params_t vae_tiling_params;
} sd_ltx_upscale_video_params_t;

typedef struct sd_ctx_t sd_ctx_t;
struct ggml_tensor;

Expand Down Expand Up @@ -512,6 +522,11 @@ SD_API bool generate_video(sd_ctx_t* sd_ctx,
sd_image_t** frames_out,
int* num_frames_out,
sd_audio_t** audio_out);
SD_API void sd_ltx_upscale_video_params_init(sd_ltx_upscale_video_params_t* params);
SD_API bool ltx_upscale_video(sd_ctx_t* sd_ctx,
const sd_ltx_upscale_video_params_t* params,
sd_image_t** frames_out,
int* num_frames_out);

typedef struct upscaler_ctx_t upscaler_ctx_t;

Expand Down
39 changes: 36 additions & 3 deletions src/conditioning/conditioner.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -2978,15 +2978,48 @@ struct LTXAVEmbedder : public Conditioner {
std::shared_ptr<GemmaTokenizer> tokenizer;
std::shared_ptr<LLM::LLMRunner> llm;
std::shared_ptr<LTXAVTextProjectionRunner> projector;
std::string projector_prefix;
bool dual_projection = false;

// LTX 2.5 bundles the aggregate embeds inside the Gemma 4 text-encoder checkpoint, where the
// loader has already applied the llm prefix; LTX 2.3 ships them as a separate connectors file.
static std::string resolve_projector_prefix(const String2TensorStorage& tensor_storage_map,
const std::string& llm_prefix,
const std::string& default_prefix) {
std::string bundled = llm_prefix + "." + default_prefix;
if (tensor_storage_map.find(bundled + ".video_aggregate_embed.weight") != tensor_storage_map.end()) {
return bundled;
}
return default_prefix;
}

// Gemma 4 keeps a per-layer output scalar that no Gemma 3 checkpoint has, and widens its
// full-attention heads to 512 so their q_proj is twice a sliding layer's.
static LLM::LLMArch detect_gemma_arch(const String2TensorStorage& tensor_storage_map,
const std::string& llm_prefix) {
if (tensor_storage_map.find(llm_prefix + ".model.layers.0.layer_scalar") != tensor_storage_map.end()) {
return LLM::LLMArch::GEMMA4_12B;
}
auto global_q = tensor_storage_map.find(llm_prefix + ".model.layers.5.self_attn.q_proj.weight");
auto sliding_q = tensor_storage_map.find(llm_prefix + ".model.layers.0.self_attn.q_proj.weight");
if (global_q != tensor_storage_map.end() &&
sliding_q != tensor_storage_map.end() &&
global_q->second.ne[1] == sliding_q->second.ne[1] * 2) {
return LLM::LLMArch::GEMMA4_12B;
}
return LLM::LLMArch::GEMMA3_12B;
}

LTXAVEmbedder(ggml_backend_t backend,
const String2TensorStorage& tensor_storage_map = {},
const std::string& llm_prefix = "text_encoders.llm",
const std::string& projector_prefix = "text_embedding_projection",
const std::string& default_projector_prefix = "text_embedding_projection",
std::shared_ptr<RunnerWeightManager> weight_manager = nullptr) {
LLM::LLMArch arch = detect_gemma_arch(tensor_storage_map, llm_prefix);
projector_prefix = resolve_projector_prefix(tensor_storage_map, llm_prefix, default_projector_prefix);
LOG_INFO("ltxav text encoder: %s", arch == LLM::LLMArch::GEMMA4_12B ? "gemma 4" : "gemma 3");
tokenizer = std::make_shared<GemmaTokenizer>();
llm = std::make_shared<LLM::LLMRunner>(LLM::LLMArch::GEMMA3_12B,
llm = std::make_shared<LLM::LLMRunner>(arch,
backend,
tensor_storage_map,
llm_prefix,
Expand All @@ -3001,7 +3034,7 @@ struct LTXAVEmbedder : public Conditioner {

void get_param_tensors(std::map<std::string, ggml_tensor*>& tensors) override {
llm->get_param_tensors(tensors, "text_encoders.llm");
projector->get_param_tensors(tensors, "text_embedding_projection");
projector->get_param_tensors(tensors, projector_prefix);
}

void get_param_tensor_ops(std::map<ggml_tensor*, enum ggml_op>& tensor_ops) override {
Expand Down
7 changes: 4 additions & 3 deletions src/model/common/block.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -268,10 +268,11 @@ class FeedForward : public GGMLBlock {
int64_t dim_out,
int64_t mult = 4,
Activation activation = Activation::GEGLU,
bool precision_fix = false) {
bool precision_fix = false,
bool bias = true) {
int64_t inner_dim = dim * mult;
if (activation == Activation::GELU) {
blocks["net.0"] = std::shared_ptr<GGMLBlock>(new GELU(dim, inner_dim));
blocks["net.0"] = std::shared_ptr<GGMLBlock>(new GELU(dim, inner_dim, bias));
} else {
blocks["net.0"] = std::shared_ptr<GGMLBlock>(new GEGLU(dim, inner_dim));
}
Expand All @@ -285,7 +286,7 @@ class FeedForward : public GGMLBlock {
// The purpose of the scale here is to prevent NaN issues in certain situations.
// For example, when using Vulkan without enabling force_prec_f32,
// or when using CUDA but the weights are k-quants.
blocks["net.2"] = std::shared_ptr<GGMLBlock>(new Linear(inner_dim, dim_out, true, false, force_prec_f32, scale));
blocks["net.2"] = std::shared_ptr<GGMLBlock>(new Linear(inner_dim, dim_out, bias, false, force_prec_f32, scale));
}

ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x) {
Expand Down
Loading