Skip to content

cookbook(diffusers): add FP8 quantized checkpoint support - #304

Closed
ConstBob wants to merge 11 commits into
NVIDIA:mainfrom
ConstBob:feat/diffusers-fp8
Closed

cookbook(diffusers): add FP8 quantized checkpoint support#304
ConstBob wants to merge 11 commits into
NVIDIA:mainfrom
ConstBob:feat/diffusers-fp8

Conversation

@ConstBob

@ConstBob ConstBob commented Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The cookbook content has moved out of this repo.
Please review the migrated changes here instead:
NVIDIA/cosmos-framework#298

Adds FP8 quantized checkpoint support to the Cosmos3 Diffusers cookbook for FP8 models.

ConstBob and others added 2 commits August 5, 2026 14:02
@ConstBob
cookbook(diffusers): add FP8 quantized checkpoint loading path
39b17e0 
Enable Nano/Super and Super 4-Step FP8 via COSMOS3_DIFFUSERS_PRECISION
without duplicating request cells. Load public MODEL_IDS with revision=fp8
or a local COSMOS3_FP8_MODEL_PATH.
@ConstBob
Merge branch 'NVIDIA:main' into feat/diffusers-fp8
8d34c6c
@ConstBob
cookbook(diffusers): fix Distilled FP8 ModelOpt restore via transform…
f4878b6 
…er preload

Modular load_components passes repo+subfolder, which ModelOpt 0.44 ignores
when looking up modelopt_state.pth. Pre-load transformer from its full local
path, pin it, then load remaining components. Add verify_fp8 helper.
@ConstBob
ConstBob force-pushed the feat/diffusers-fp8 branch from 1831c57 to f4878b6 Compare August 6, 2026 18:29
@ConstBob
cookbook(diffusers): point FP8 loads at dedicated Hub repos
619afb8 
Load Nano/Super/Edge FP8 from nvidia/Cosmos3-*-FP8 and keep Super 4-Step on revision=fp8. Rebase the notebook onto current audiovisual helpers without documenting mixed precision.
@ConstBob
Merge NVIDIA/cosmos main into feat/diffusers-fp8
a3bd513 
# Conflicts:
#	cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb
@ConstBob
cookbook(diffusers): mention checkpoint mixed W8A8/W8A16 schedule
c4ecc2d 
Point users at the Cosmos3 Diffusers docs instead of duplicating generate-time kwargs.
@ConstBob
Merge NVIDIA/cosmos main into feat/diffusers-fp8
86a24b4
@ConstBob
cookbook(diffusers): load action and transfer FP8 from dedicated Hub …
b8fdd60 
…repos

Point forward dynamics, inverse dynamics, and video transfer at the Nano and Super FP8 model cards without changing their request cells.
@ConstBob
cookbook(diffusers): fix pose_utils import in ID notebook
9ed508b
@ConstBob
ConstBob requested a review from yzhautouskay October 17, 2026 00:13
yzhautouskay
yzhautouskay suggested changes Oct 19, 2026
View reviewed changes
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb Resolved
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb Resolved
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb Resolved
Comment thread cookbooks/cosmos3/generator/action/run_fd_with_diffusers.ipynb Resolved
@ConstBob

ConstBob commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed all review comments in dd7088e:

  • Explicitly fetches transformer/modelopt_state.pth before regular FP8 Hub loads and validates local checkpoint paths.
  • Automatically verifies FP8 weights, restored ModelOpt quantizers, and absence of meta tensors.
  • Fixes the distilled payload summary KeyError.
  • Removes the outdated private-repository notes.
  • Keeps non-transformer modular components on their configured BF16 sources, avoiding the FP8 repository’s root Reasoner state being applied to the VAE or audio tokenizer.

Validation completed:

  • Fresh-cache Nano FP8 restore: weights=504, quantizers=1533, meta=0
  • Nano FP8 35-step 720p T2I
  • Nano FP8 Action FD and ID
  • Nano FP8 Transfer: 121 frames, 50 steps
  • Distilled Super 4-Step FP8: weights=896, quantizers=2709, meta=0, followed by successful T2I generation
  • Notebook schema validation and Ruff checks passed

@yzhautouskay, could you please take another look when convenient?

@ConstBob
ConstBob requested a review from yzhautouskay October 19, 2026 18:47
yzhautouskay
yzhautouskay approved these changes Oct 20, 2026
View reviewed changes

@yzhautouskay yzhautouskay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@ConstBob ConstBob mentioned this pull request Oct 21, 2026
Merged
@yaoxu-crypto

yaoxu-crypto commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Closing this PR, using NVIDIA/cosmos-framework#298

christinayyw added a commit to NVIDIA/cosmos-framework that referenced this pull request Nov 5, 2026
@ConstBob @christinayyw
cookbook(diffusers): add FP8 quantized checkpoint support (#298)
d7d2e31 
> **Do not merge.** Wait for legal approval on the FP8 Hugging Face
model cards. The model cards need to be approved and published on
Hugging Face before this GitHub PR is merged.

## Summary
Ports the Diffusers FP8 cookbook support from NVIDIA/cosmos#304 to
`NVIDIA/cosmos-framework` following the cookbook repository
reorganization.
Updates the audiovisual, action forward/inverse dynamics, and transfer
notebooks to:
- select BF16 or FP8 through `COSMOS3_DIFFUSERS_PRECISION`
- load Nano, Super, and Edge from their dedicated FP8 repositories
- load distilled T2I/I2V checkpoints from their `fp8` revisions
- enable ModelOpt checkpoint restoration and FP8 GEMM support
- explicitly fetch `transformer/modelopt_state.pth` for a fresh Hub
cache
- preload modular transformers where required by ModelOpt 0.44
- validate FP8 weights, quantizers, and unresolved meta tensors after
loading
The existing BF16 behavior remains the default.
## Repository migration
The original implementation and review history are in NVIDIA/cosmos#304.
After NVIDIA/cosmos#384 and #292 moved cookbook
content into `.agents/skills/`, the changes were rebased onto the new
Framework locations while preserving the current Framework setup,
`uv 0.11.29` guidance, and updated action imports.
## Validation
Validated on an NVIDIA RTX PRO 6000 Blackwell:
- Nano FP8 from an empty Hub cache:
  - ModelOpt state restored
  - 504 FP8 weights, 1533 quantizers, 0 meta tensors
  - complete 189-frame, 1280×720 T2V generation
- Super FP8 from an independent empty cache:
  - 896 FP8 weights, 2709 quantizers, 0 meta tensors
- Distilled Super T2I FP8:
  - complete 768×768 image generation
- Distilled Super I2V FP8:
  - complete 189-frame, 832×480 video generation
- Nano action forward dynamics, inverse dynamics, and transfer paths
were
  exercised end to end in the original implementation.
- Generated videos were decoded and checked for expected frame count and
  resolution.

---------

Co-authored-by: Christina Wang <77945617+christinayyw@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@yzhautouskay yzhautouskay yzhautouskay approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ConstBob @yaoxu-crypto @yzhautouskay