Repository navigation
LMSDiscreteScheduler uses the wrong multistep order when sampling starts part-way through the schedule (img2img, inpainting) #15108
Description
Describe the bug
LMSDiscreteScheduler.step sets the multistep order from step_index:
diffusers/src/diffusers/schedulers/scheduling_lms_discrete.py
Lines 605 to 612 in 73faca3
step_index is the position in the full schedule. Image-to-image and inpainting with strength < 1 (and SDXL img2img with denoising_start) start part-way through the schedule with set_begin_index:
So the first steps use 2nd to 4th order coefficients while only one derivative is stored, and zip(lms_coeffs, reversed(self.derivatives)) drops the coefficients that have no derivative. The first update becomes sample + c0 * d instead of sample + (sigma_next - sigma) * d, and the next steps are off in the same way. The latents then sit at a different noise level than the timestep the model gets next, which leaves noise in the result. k-diffusion's sample_lms counts the order from the steps taken on the remaining sigmas.
This looks like the cause of #5630 (LMS with SDXL img2img leaves noise at low strength), which was closed by the stale bot.
I found this by reading and testing the code. I'd like to open a PR if this approach is fine: order = min(len(self.derivatives), order). Text-to-image is unchanged, because there len(self.derivatives) equals min(step_index + 1, order).
Reproduction
I didn't run a real checkpoint (CPU only here). The closest I have runs the real StableDiffusionImg2ImgPipeline with LMSDiscreteScheduler, but with a stand-in UNet that gives the exact noise prediction when the only training sample is the input latents. With an exact model, a correct sampler returns the input latents.
import math
import torch
from diffusers import LMSDiscreteScheduler, StableDiffusionImg2ImgPipeline
from diffusers.configuration_utils import ConfigMixin, register_to_config
from diffusers.models.modeling_utils import ModelMixin
class ExactUNet(ModelMixin, ConfigMixin):
# exact noise prediction when every training sample equals `self.x0`
@register_to_config
def __init__(self, in_channels: int = 4, time_cond_proj_dim=None):
super().__init__()
self.dummy = torch.nn.Parameter(torch.zeros(1))
def forward(self, sample, timestep, encoder_hidden_states=None, return_dict=False, **kwargs):
ac = self.alphas_cumprod.double()
log_sigmas = 0.5 * torch.log((1 - ac) / ac)
t = float(timestep)
lo = min(int(t), len(ac) - 1)
hi = min(lo + 1, len(ac) - 1)
sigma = torch.exp((1 - (t - lo)) * log_sigmas[lo] + (t - lo) * log_sigmas[hi]).item()
alpha = 1 / math.sqrt(1 + sigma**2)
return ((sample - alpha * self.x0) / (sigma * alpha),)
scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", steps_offset=1)
unet = ExactUNet()
pipe = StableDiffusionImg2ImgPipeline(
vae=None,
text_encoder=None,
tokenizer=None,
unet=unet,
scheduler=scheduler,
safety_checker=None,
feature_extractor=None,
requires_safety_checker=False,
)
pipe.set_progress_bar_config(disable=True)
unet.alphas_cumprod = pipe.scheduler.alphas_cumprod
torch.manual_seed(1)
x0 = torch.randn(1, 4, 32, 32) * 0.8 # passed to the pipeline as latents
unet.x0 = x0
for steps, strength in [(10, 0.3), (10, 0.5), (25, 0.3), (25, 0.5), (50, 0.3)]:
out = pipe(
prompt_embeds=torch.zeros(1, 77, 32),
image=x0,
strength=strength,
num_inference_steps=steps,
guidance_scale=1.0,
output_type="latent",
generator=torch.Generator().manual_seed(0),
).images
print(f"steps={steps} strength={strength} relative error={((out - x0).norm() / x0.norm()).item():.4f}")| steps | strength | main (73faca3) | with order = min(len(self.derivatives), order) |
|---|---|---|---|
| 10 | 0.3 | 0.0824 | 0.0000 |
| 10 | 0.5 | 0.0321 | 0.0000 |
| 25 | 0.3 | 0.0630 | 0.0000 |
| 25 | 0.5 | 0.0019 | 0.0000 |
| 50 | 0.3 | 0.0002 | 0.0000 |
I also compared the same scheduler loop with k-diffusion's sample_lms run on the remaining sigmas (Gaussian-data optimal denoiser, 10/25/50 steps, strength 0.3/0.5/0.75, SD 1.5 and SDXL scheduler settings). On main the outputs differ by 0.4% to 29% (max abs difference over max abs value); with the change they are identical.
Logs
main (73faca3):
steps=10 strength=0.3 relative error=0.0824
steps=10 strength=0.5 relative error=0.0321
steps=25 strength=0.3 relative error=0.0630
steps=25 strength=0.5 relative error=0.0019
steps=50 strength=0.3 relative error=0.0002System Info
diffusers 0.42.0.dev0 (main @ 73faca3), Python 3.12.13, torch 2.14.1, Linux, CPU only.
Who can help?
I made this report and directed the investigation, with AI assistance from Claude and GPT-6 Astra.
Activity
mottopanikeiku commentedon Oct 24, 2026
for reference, my fix has been ready since i opened this issue: mottopanikeiku@199d1d4 (committed oct 22). it's the change proposed above, a
test_full_loop_with_begin_indexregression test (max error 4.1e-2 before, 1.2e-7 after) and the updatedtest_full_loop_with_noisevalues. i held off on the pr because CONTRIBUTING asks for a maintainer go-ahead on the issue first. #15116 makes the same change and uses the reproducer numbers from this issue.@yiyixuxu @dg845 i can open mine as soon as you're ok with the approach.