Repository navigation
Conversation
…step with final_sigmas_type=sigma_min When final_sigmas_type='sigma_min' is combined with schedules whose last step already reaches sigma_min (e.g., use_karras_sigmas, use_exponential_sigmas, use_beta_sigmas, or use_lu_lambdas), the appended terminal sigma duplicates the penultimate sigma, producing h = lambda_t - lambda_s = 0.0 on the final step and 0.0 / 0.0 = NaN in higher-order updates. Fall back to the first-order update on the final step when sigmas[step_index + 1] == sigmas[step_index]. Fixes huggingface#14890 Signed-off-by: Parth Vadera <parthvadera@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
|
Gentle, polite ping @stevhliu @sayakpaul @yiyixuxu @DN6 @hlky — whenever you have a spare moment, would you mind taking a quick look at this small, self-contained scheduler fix for #14890? When |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #14890.
Root Cause
When
final_sigmas_type="sigma_min"is enabled alongside sigma schedules whose final step already reachessigma_min(use_karras_sigmas=True,use_exponential_sigmas=True,use_beta_sigmas=True, oruse_lu_lambdas=True),set_timesteps()appendssigma_last = sigmas[-1]to the sigma array, making the last two entries identical (sigmas[-2] == sigmas[-1]).On the final denoising step (
step_index == num_inference_steps - 1), this produces a zero-width log-SNR step (h = lambda_t - lambda_s == 0.0). Whiledpm_solver_first_order_updatesafely multiplies byexpm1(-h) = 0.0(leaving the sample unchanged), the 2nd-order and 3rd-order multistep updates inDPMSolverMultistepScheduler(and the multistep predictor/corrector inUniPCMultistepScheduler) divideh = 0.0byr0 = h_0 / h = inf(or computehh / h_0whereh_0 = 0.0), producing0.0 / 0.0 = NaNand corrupting the entire output tensor on the very last step.Solution
DPMSolverMultistepScheduler.step(): include(self.step_index == self.num_inference_steps - 1 and self.sigmas[self.step_index + 1] == self.sigmas[self.step_index])inlower_order_finalso zero-width terminal steps usedpm_solver_first_order_update.UniPCMultistepScheduler.step(): bypass the multistep corrector when the previous step had zero width (self.sigmas[self.step_index] == self.sigmas[self.step_index - 1]) and fall back to first-ordermultistep_uni_p_bh_update(..., order=1)when the final step has zero width (self.sigmas[self.step_index + 1] == self.sigmas[self.step_index]).test_zero_width_final_stepin bothtests/schedulers/test_scheduler_dpm_multi.pyandtests/schedulers/test_scheduler_unipc.py.