LTX-2.5 spatial and temporal upscaling only mode - #1921
Draft
jdluzen wants to merge 2 commits into
Draft
Conversation
LTX-2.5 shares its video VAE, audio VAE, vocoder and spatial upscaler architecture with LTX-2.3, so those need no changes. The transformer differs in two places and the text encoder moves from Gemma 3 to Gemma 4. Transformer: - Detect ff_bias and audio_ff_bias separately from the weights. LTX-2.5 sets ff_bias=false but leaves audio_ff_bias at its default, so a single flag would drop the audio FFN biases. - Add the keyframes_abs_pos_embedding marker, applied after patchify_proj to the tokens of the first latent frame. - Cast learnable_registers to F32 when the checkpoint stores them at another type, as the published LTX-2.5 GGUFs do. Text encoder (LLMArch::GEMMA4_12B): - Full-attention layers use a 512-wide head with a single KV head and no v_proj, taking V from the raw K projection; sliding layers keep 256/8. - V is normalized with a weightless RMS norm and never rotated. - Attention runs unscaled, and RoPE is proportional on full-attention layers, implemented with freq_factors so the unrotated pairs collapse to identity without reordering the head. - Each layer ends with its learned layer_scalar. - Both the transformers and the llama.cpp GGUF layer-norm namings are accepted, so the bundled single-file checkpoint loads directly. The Gemma 4 vocabulary differs from Gemma 3 only in reserved <unusedN> slots and shares its merge table, so the embedded Gemma vocab still produces the right token ids. LTX-2.5 bundles the text projection into its text encoder, so --embeddings-connectors is no longer required; the projection is looked up under the --llm prefix as well. Verified on the Q8_0 dev transformer for T2V, I2V, spatial latent upscaling and a Q8_0-converted text encoder, and against Z-Image for the shared attention and feed-forward changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Based on #1893, this adds a standalone spatial and temporal upscaling only mode using LTX-2.5.
Related Issue / Discussion
#1893 should merge in first.
Additional Information
Completes successfully from some random frames I have. Only CPU is tested currently, need to test other backends.
Keeping this as draft only for now.
Checklist