TripoSplat

docs/source/en/_toctree.yml

@@ -433,6 +433,8 @@
         title: Transformer2DModel
       - local: api/models/transformer_temporal
         title: TransformerTemporalModel
+      - local: api/models/triposplat
+        title: TripoSplat models
       - local: api/models/wan_animate_2_transformer_3d
         title: WanAnimate2Transformer3DModel
       - local: api/models/wan_animate_transformer_3d
@@ -701,6 +703,8 @@
         title: Stable Diffusion
       - local: api/pipelines/stable_unclip
         title: Stable unCLIP
+      - local: api/pipelines/triposplat
+        title: TripoSplat
       - local: api/pipelines/value_guided_sampling
         title: Value-guided sampling
       - local: api/pipelines/visualcloze

docs/source/en/api/models/triposplat.md

@@ -0,0 +1,43 @@
+# TripoSplat models
+
+[TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) generates 3D Gaussian splats from an image. Its transformer
+jointly denoises Gaussian and camera latents, conditioned on DINOv3 features and packed Flux2 VAE image latents.
+The Gaussian decoder samples an octree and predicts splat parameters at the requested density.
+
+## TripoSplatTransformer3DModel
+
+[[autodoc]] TripoSplatTransformer3DModel
+    - all
+    - forward
+
+## TripoSplatGaussianDecoder
+
+[[autodoc]] TripoSplatGaussianDecoder
+    - all
+    - forward
+
+The decoder returns a tensor of shape `(batch, num_gaussians, 14)`. Columns contain xyz position (3), degree-zero
+spherical harmonic color (3), positive scale (3), wxyz rotation (4), and opacity (1). Positions use the model's
+coordinate system; export helpers apply the reference viewer transform.
+
+The decoder's adaptive octree uses data-dependent tensor sizes. Compile its repeated transformer blocks with
+`decoder.compile_repeated_blocks()`; compiling the whole decoder as a static graph is not supported.
+
+## Original single-file checkpoints
+
+The transformer, Gaussian decoder, and optional `BiRefNetModel` support `from_single_file`. Supply the component's
+local Diffusers config explicitly; a default converted Hub repository has not been published yet.
+
+```python
+import torch
+from diffusers import TripoSplatTransformer3DModel
+
+transformer = TripoSplatTransformer3DModel.from_single_file(
+    "./ckpts/diffusion_models/triposplat_fp16.safetensors",
+    config="./triposplat-diffusers/transformer",
+    dtype=torch.float16,
+)
+```
+
+Use the same loading method with `TripoSplatGaussianDecoder` and its `decoder` config, or `BiRefNetModel` and its
+`background_remover` config. The checkpoint key mappings are shared with the conversion script.

docs/source/en/api/pipelines/triposplat.md

@@ -0,0 +1,119 @@
+# TripoSplat
+
+[TripoSplat](https://github.com/VAST-AI-Research/TripoSplat) generates 3D Gaussian splats from foreground images.
+Both the standard and modular APIs use the same converted components. DINOv3 and the image VAE use bfloat16 in
+the reference workflow; the transformer, Gaussian decoder, and optional BiRefNet model use float16.
+
+The pipeline uses `FlowMatchEulerDiscreteScheduler` with `shift=3.0` and supplies the reference sigma grid.
+Predictions are cast to float32 for scheduler updates so Gaussian and camera latents remain in float32 throughout
+denoising. Sigma storage and velocity updates use float32, which can produce numerical differences from the reference.
+
+## Convert the checkpoint
+
+The [official checkpoint](https://huggingface.co/VAST-AI/TripoSplat) must first be converted to Diffusers format.
+Run the conversion script from a Diffusers checkout:
+
+```bash
+python scripts/convert_triposplat_to_diffusers.py \
+    --output_dir ./triposplat-diffusers \
+    --include_background_remover
+```
+
+The default `--dtype auto` preserves the reference component dtypes. An explicit `--dtype float16`,
+`--dtype bfloat16`, or `--dtype float32` applies that dtype to every component.
+
+Omit `--include_background_remover` to use images with a foreground alpha mask, or prepared RGB images with
+`is_preprocessed=True`. BiRefNet is an optional component implemented under the TripoSplat pipeline package and
+requires torchvision for deformable convolutions. It loads through the normal Diffusers model loading API.
+Its Swin attention uses additive relative-position bias, so select a backend that supports additive masks, such
+as native PyTorch SDPA.
+
+## Standard pipeline
+
+```python
+import torch
+from PIL import Image
+from diffusers import TripoSplatPipeline
+from diffusers.utils import export_to_gaussian_ply, export_to_splat
+
+dtype = {"default": torch.float16, "image_encoder": torch.bfloat16, "vae": torch.bfloat16}
+pipe = TripoSplatPipeline.from_pretrained("./triposplat-diffusers", dtype=dtype).to("cuda")
+output = pipe(
+    image=Image.open("object.png"),
+    num_inference_steps=20,
+    guidance_scale=3.0,
+    num_gaussians=32768,
+    generator=torch.Generator("cuda").manual_seed(42),
+    decoder_generator=torch.Generator("cuda").manual_seed(1234),
+)
+export_to_gaussian_ply(output.gaussians[0], "object.ply")
+export_to_splat(output.gaussians[0], "object.splat")
+```
+
+Use `num_gaussians=[32768, 65536]` to decode several densities after one denoising run. The output contains one
+batched Gaussian tensor per requested density. Counts range from 32768 to 262144 and are rounded to multiples of
+32 for the released decoder. Set `output_type="latent"` to return the denoised Gaussian and camera latents.
+
+Tensor image inputs use `(channels, height, width)` or `(batch, channels, height, width)` layout. NumPy image
+arrays use channel-last layout. Float image inputs use values between zero and one.
+
+`TripoSplatImageProcessor.prepare_foreground()` crops RGBA foregrounds and returns RGB PIL images on black
+canvases. The pipeline predicts missing alpha masks with the optional BiRefNet component before this step.
+The processor inherits `VaeImageProcessor.preprocess()` to convert the prepared RGB images to tensors in `[0, 1]`;
+the DINO and VAE encoding steps apply their respective normalization.
+
+## Modular pipeline
+
+```python
+import torch
+from PIL import Image
+from diffusers import ModularPipeline
+
+pipe = ModularPipeline.from_pretrained("./triposplat-diffusers")
+pipe.load_components(dtype={"default": torch.float16, "image_encoder": torch.bfloat16, "vae": torch.bfloat16})
+pipe.to("cuda")
+pipe.update_components(guider=pipe.guider.new(guidance_scale=3.0))
+state = pipe(
+    image=Image.open("object.png"),
+    num_inference_steps=20,
+    num_gaussians=32768,
+    generator=torch.Generator("cuda").manual_seed(42),
+    decoder_generator=torch.Generator("cuda").manual_seed(1234),
+)
+gaussians = state.get("gaussians")
+```
+
+The default blockset has `preprocess`, `image_encoder`, `vae_encoder`, `denoise`, and `decode` steps.
+The denoise sequence separates conditioning expansion, timestep setup, noise preparation, and the denoising loop.
+Each step can be removed or run independently. For example, remove `decode` from a `TripoSplatAutoBlocks`
+instance before calling `init_pipeline()` to obtain latents, then reuse them with the decode block at different
+densities. Configure guidance on the guider component.
+
+Updated `from_config` components, including `image_processor`, are not stored in the modular index by the current
+framework. After reloading, reapply a customized processor with `update_components(image_processor=...)`.
+The default 1024-pixel processor is unaffected.
+
+`TripoSplatClassifierFreeGuidance` evaluates guidance as `scale * conditional - (scale - 1) * unconditional`.
+The standard guider uses an equivalent formula with a different operation order, which changes float16 rounding.
+TripoSplat uses only the conditional prediction for guidance scales at or below one.
+
+## TripoSplatPipeline
+
+[[autodoc]] TripoSplatPipeline
+    - all
+    - __call__
+
+## TripoSplatPipelineOutput
+
+[[autodoc]] TripoSplatPipelineOutput
+
+## TripoSplatModularPipeline
+
+[[autodoc]] TripoSplatModularPipeline
+    - all
+
+## BiRefNetModel
+
+[[autodoc]] BiRefNetModel
+    - all
+    - forward

scripts/convert_triposplat_to_diffusers.py

@@ -0,0 +1,148 @@
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import argparse
+from pathlib import Path
+
+import torch
+from accelerate import init_empty_weights
+from huggingface_hub import snapshot_download
+from safetensors.torch import load_file
+from transformers import DINOv3ViTConfig, DINOv3ViTModel
+
+from diffusers import (
+    AutoencoderKLFlux2,
+    FlowMatchEulerDiscreteScheduler,
+    TripoSplatAutoBlocks,
+    TripoSplatPipeline,
+)
+from diffusers.loaders.single_file_utils import (
+    convert_triposplat_birefnet_checkpoint_to_diffusers,
+    convert_triposplat_decoder_checkpoint_to_diffusers,
+    convert_triposplat_transformer_checkpoint_to_diffusers,
+)
+from diffusers.models.autoencoders.autoencoder_triposplat import TripoSplatGaussianDecoder
+from diffusers.models.transformers.transformer_triposplat import TripoSplatTransformer3DModel
+from diffusers.pipelines.triposplat.modeling_birefnet import BiRefNetModel
+
+
+def _load_converted(model_class, state_dict, **kwargs):
+    with init_empty_weights():
+        model = model_class(**kwargs)
+    model.load_state_dict(state_dict, strict=True, assign=True)
+    return model.eval()
+
+
+def convert_triposplat_transformer(state_dict):
+    return _load_converted(
+        TripoSplatTransformer3DModel, convert_triposplat_transformer_checkpoint_to_diffusers(state_dict)
+    )
+
+
+def convert_triposplat_decoder(state_dict):
+    return _load_converted(TripoSplatGaussianDecoder, convert_triposplat_decoder_checkpoint_to_diffusers(state_dict))
+
+
+def convert_triposplat_birefnet(state_dict):
+    return _load_converted(BiRefNetModel, convert_triposplat_birefnet_checkpoint_to_diffusers(state_dict))
+
+
+def convert_triposplat_image_encoder(state_dict):
+    config = DINOv3ViTConfig(
+        hidden_size=1280,
+        intermediate_size=5120,
+        num_hidden_layers=32,
+        num_attention_heads=20,
+        patch_size=16,
+        num_register_tokens=4,
+        layer_norm_eps=1e-5,
+        query_bias=True,
+        key_bias=False,
+        value_bias=True,
+        use_gated_mlp=True,
+        hidden_act="silu",
+        rope_theta=100.0,
+    )
+    with init_empty_weights():
+        model = DINOv3ViTModel(config)
+    target_keys = model.state_dict()
+    converted = {
+        (f"model.{key}" if key.startswith("layer.") and f"model.{key}" in target_keys else key): value
+        for key, value in state_dict.items()
+    }
+    model.load_state_dict(converted, strict=True, assign=True)
+    return model.eval()
+
+
+def convert_triposplat_image_vae(state_dict):
+    return _load_converted(AutoencoderKLFlux2, state_dict, batch_norm_eps=1e-5)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Convert the official TripoSplat inference checkpoints to Diffusers.")
+    parser.add_argument("--checkpoint_dir", type=Path)
+    parser.add_argument("--output_dir", type=Path, required=True)
+    parser.add_argument("--revision", default="56a96e603204ec410c4da60c13ea4fa09a2169a9")
+    parser.add_argument(
+        "--dtype",
+        choices=("auto", "float32", "float16", "bfloat16"),
+        default="auto",
+        help="auto uses the reference component dtypes; an explicit dtype applies to every component.",
+    )
+    parser.add_argument("--include_background_remover", action="store_true")
+    args = parser.parse_args()
+    files = {
+        "transformer": ("diffusion_models/triposplat_fp16.safetensors", convert_triposplat_transformer),
+        "decoder": ("vae/triposplat_vae_decoder_fp16.safetensors", convert_triposplat_decoder),
+        "image_encoder": ("clip_vision/dino_v3_vit_h.safetensors", convert_triposplat_image_encoder),
+        "vae": ("vae/flux2-vae.safetensors", convert_triposplat_image_vae),
+    }
+    if args.include_background_remover:
+        files["background_remover"] = ("background_removal/birefnet.safetensors", convert_triposplat_birefnet)
+    checkpoint_dir = args.checkpoint_dir
+    if checkpoint_dir is None:
+        checkpoint_dir = Path(
+            snapshot_download(
+                "VAST-AI/TripoSplat", revision=args.revision, allow_patterns=[item[0] for item in files.values()]
+            )
+        )
+    components = {}
+    for name, (filename, convert) in files.items():
+        model = convert(load_file(str(checkpoint_dir / filename)))
+        if args.dtype == "auto":
+            dtype = torch.bfloat16 if name in ("image_encoder", "vae") else torch.float16
+        else:
+            dtype = getattr(torch, args.dtype)
+        model.to(dtype=dtype).save_pretrained(args.output_dir / name)
+        reloaded = type(model).from_pretrained(args.output_dir / name, torch_dtype=dtype)
+        if set(model.state_dict()) != set(reloaded.state_dict()):
+            raise RuntimeError(f"The {name} checkpoint did not round-trip.")
+        for key, value in model.state_dict().items():
+            if not torch.equal(value, reloaded.state_dict()[key]):
+                raise RuntimeError(f"The {name} checkpoint changed {key} on reload.")
+        components[name] = reloaded
+        del model
+        print(f"Converted {name}", flush=True)
+    components.setdefault("background_remover", None)
+    pipeline = TripoSplatPipeline(**components, scheduler=FlowMatchEulerDiscreteScheduler(shift=3.0))
+    pipeline.save_pretrained(args.output_dir)
+    modular = TripoSplatAutoBlocks().init_pipeline()
+    modular.update_components(**pipeline.components)
+    modular.save_pretrained(str(args.output_dir), overwrite_modular_index=True)
+    print(f"Saved standard and modular pipelines to {args.output_dir}", flush=True)
+
+
+if __name__ == "__main__":
+    main()