Prompt2 - #518
Open
3a1b2c3 wants to merge 25 commits into
Open
Conversation
…che) Rebuild the per-block cross-attention text KV in place at a chunk boundary (storage addresses survive, so captured CUDA graphs stay valid) while the self-attention history carries the scene forward under the new prompt. Optional two-prompt edit guidance runs the cond branch under old and new text against the same history and extrapolates flow_old + s*(flow_new - flow_old) for N chunks after a swap; the KV commit always runs single-branch under the new prompt. ReCache (LongLive / Hunyuan-GameCraft-2) re-commits the previous chunk's KV under the new text via a same-index cache-bracket re-open. GPU-verified: swaps are RNG-clean (zero pre-swap divergence), and weather/lighting edits land convincingly at s=3 with training-caption-style phrasing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route datachannel event messages to a free-text prompt swap (event_id carries the prompt; clear states restore the scene prompt) and to /spawn <preset> [dist] [speed] [lateral] / /clear-actors commands. Spawned actors follow a constant-velocity world trajectory and enter the conditioning through the same Ludus bbox path as gRPC dynamic actors, so the model materializes grounded vehicles/pedestrians the game shell can track. The web client gains a scene-prompt panel and spawn buttons; WASD typed into text fields no longer drives the car. GPU-verified: a spawned car materializes photorealistically within one chunk and vanishes within one chunk of /clear-actors. Guidance defaults (s=3, 6 chunks) follow the calibration sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
smoke_text_edit rolls RNG-matched control/swap/guided/recache variants and reports per-chunk divergence; sweep_text_edit calibrates an edit prompt bank (incl. the scene bundle's native weather phrasings) against one control; smoke_spawn_actor drives the WebRTC runtime headless and exercises /spawn and /clear-actors. guidance_distill/PLAN.md specifies the Tier-2a LoRA recipe that bakes two-prompt edit guidance into the student (post-swap-gated, premerge-deployed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ego pose is the rig origin (~1.5 m above the road), so spawned boxes floated at eye level — off-distribution for the bbox conditioning, and the model under-rendered them (a moving truck box was ignored entirely). Offset the bbox center by the rig height; verified against the scene's own actor boxes in the rendered conditioning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/spawn <preset> [dist] [speed] [lateral] [yaw_deg] — box heading relative to the ego (0 = same direction, 180 = oncoming). The rendered box's front/back face colors encode travel direction. Probing found the model paints static boxes in place (parked-vehicle prior) but renders constant-gap moving boxes as a plausible oncoming pass regardless of yaw; the argument stays for scene priors where lead vehicles exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Distill the two-prompt text-edit guidance into a LoRA so a plain prompt swap responds at guided strength: on-policy rollouts with a mid-stream swap, teacher = the frozen base running the guidance combine on the same states, student = the LoRA'd single branch under the new prompt (per-term immediate backward under functional attention — the teacher's in-place KV loads would otherwise invalidate the student's checkpoint recompute). Prompt bank reuses the calibration-sweep phrasings plus no-op swaps as a drift regularizer; embeddings precomputed so the 14 GB text encoder is not resident during training. r64 / 1600 steps passes the eval gate on held-out clips: the LoRA'd plain swap reaches 0.854 of guided divergence (bar 0.8; base 0.376), visually clean, generalizing across scene types. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
omnidreams/_edit_lora.py caches base and base-plus-delta weight sets at load and toggles them by in-place copy_ at edit-window boundaries, so weight storage addresses survive and captured CUDA graphs stay valid (the drift corrector's pointer-rebinding swap is not graph-safe). With the hook attached (text_edit_lora_path on the wrapper / WebRTC runtime config, EDIT_LORA on the probe script), replace_text_embeddings builds a use_lora window: single forward per denoise step at guided strength, KV commits included, base weights restored on expiry and on new rollouts. GPU-validated on the rain benchmark: plain swaps stay bit-identical to the hookless run, and the LoRA window's divergence curve tracks the two-branch guided reference (same endpoint) with zero extra forwards — replacing the +84 ms/chunk guidance cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up (PR NVIDIA#431): the ReCache context forward drew its noise from the model RNG, so enabling ReCache shifted every subsequent noise draw relative to a plain-swap rollout. Any noise rendition of the same clean latent is in-distribution for the context forward (each chunk's original commit already uses an independent draw), but drawing from a per-AR-index seeded generator makes the re-commit deterministic and leaves the rollout's noise stream untouched with or without ReCache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Main moved the runner video helpers to flashdreams.infra.runner_io (load_video_tensor / load_first_frame_tensor / write_video_tensor) — port the three GPU probe scripts to the new API. The guidance_distill trainer/eval/precompute scripts import the Clean Forcing training infra (drift_correction/), which is not on main yet — keep PLAN.md here and land the scripts with that stack (NVIDIA#398); the deploy hook (_edit_lora.py) is self-contained and stays. Also satisfy ty: cast the torch.compile unwrap, require the test checkpoint path, annotate the two intentional test monkeypatches; ruff-format the touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Greptile SummaryThe PR adds mid-stream prompt editing across Omnidreams local and WebRTC sessions, Windows setup/workarounds, and a new Cosmos LoRA workflow. It also expands diagnostics, smoke tests, launch scripts, and interactive actor controls.
Confidence Score: 2/5The PR is not safe to merge until HUD prompt updates are serialized with generation and the advertised Cosmos LoRA training and inference commands perform their stated operations. HUD edits can concurrently mutate the live autoregressive cache while a chunk is generated, and both new Cosmos LoRA user entrypoints report success without training, saving checkpoints, loading weights, or producing video. Files Needing Attention: integrations/omnidreams/omnidreams/interactive_drive/demo.py, cosmos_lora/scripts/train_lora.py, cosmos_lora/scripts/inference_lora.py Important Files Changed
Sequence DiagramsequenceDiagram
participant U as HUD User
participant P as Presenter Executor
participant W as Chunk Pipeline Worker
participant M as Omnidreams Pipeline
U->>P: Submit edited prompt
par Current changed path
P->>M: replace_text(live cache)
and Chunk generation
W->>M: generate(live cache)
end
Note over P,W: Cache access is not serialized
Reviews (1): Last reviewed commit: "style" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # Try replace_text first (if text encoder is loaded) | ||
| if hasattr(pipeline, "text_encoder") and pipeline.text_encoder is not None: | ||
| logger.info(f"[demo] calling pipeline.replace_text() with prompt: '{prompt[:60]}...'") | ||
| pipeline.replace_text(cache, [[prompt]]) |
Contributor
There was a problem hiding this comment.
When a user submits a scene prompt during chunk generation, the presenter executor calls replace_text on the live cache concurrently with the chunk-pipeline worker, violating the API's between-step requirement and causing the generated chunk and cross-attention cache to use inconsistent prompt state. The offloaded-encoder path also checks frame_queue on the wrong pipeline object, waits ten seconds, and then performs the same unsynchronized update.
Knowledge Base Used:
Comment on lines
+113
to
+129
| for batch_idx, batch in enumerate(dataloader): | ||
| # TODO: Training step | ||
| # - Load video frames | ||
| # - Encode with text prompt | ||
| # - Forward pass through model | ||
| # - Compute loss | ||
| # - Backward pass | ||
| # - Update LoRA weights | ||
|
|
||
| if (batch_idx + 1) % 10 == 0: | ||
| logger.info(f" Batch {batch_idx+1}/{len(dataloader)}") | ||
|
|
||
| # Save checkpoint | ||
| if (epoch + 1) % save_interval == 0: | ||
| checkpoint_path = output_path / f"checkpoint_epoch_{epoch+1}.pt" | ||
| logger.info(f"Saving checkpoint to {checkpoint_path}") | ||
| # TODO: Save LoRA weights |
Contributor
There was a problem hiding this comment.
When a user runs the documented LoRA training command, this loop neither initializes a model nor performs optimization or writes checkpoint files, yet it reports successful training and saved checkpoints. The advertised inference command then fails because checkpoint_epoch_N.pt was never created.
Comment on lines
+61
to
+74
| # TODO: Load model with LoRA weights | ||
| # 1. Load base Cosmos model | ||
| # 2. Load LoRA weights from checkpoint | ||
| # 3. Merge or apply LoRA adapter | ||
| logger.info("Loading model with LoRA weights (requires cosmos SDK)...") | ||
| logger.info("Run: pip install nvidia-cosmos") | ||
|
|
||
| # TODO: Generate video | ||
| logger.info(f"\nGenerating video with prompt: '{prompt}'") | ||
| logger.info("Inference not yet implemented - requires Cosmos SDK") | ||
|
|
||
| # Expected output | ||
| output_video = output_path / "generated.mp4" | ||
| logger.info(f"\nVideo would be saved to: {output_video}") |
Contributor
There was a problem hiding this comment.
When a user invokes the documented inference command with an existing checkpoint, this path reads only the neighboring configuration and never loads adapter weights, runs the model, or writes generated.mp4, so the command exits successfully without producing the promised video.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.