DEIS coefficient precision causes increasing error as inference steps increase #15090

Describe the bug

At 50 inference steps, DEIS returns about 0.01679 for an exact constant-predictor trajectory whose final value is about 0.01000. Increasing the step count makes the error much larger. I reproduced this with both float32 and float64 samples on current main.

The control casts scheduler.sigmas after set_timesteps, so every noise-grid value stays identical. The predictor always returns one and the initial sample is about one. At 50 steps, the absolute error is about 0.00679, or 68% of the expected final value.

For constant epsilon, d(x / alpha) / d(rho) = 1, where rho is the stored sigma ratio and alpha = 1 / sqrt(1 + rho**2). The script evaluates that exact solution, including the rounding of the initial sample. Lagrange interpolation reproduces constants, so this trajectory has no truncation error in exact arithmetic.

The evidence points to cancellation in the float32 scalar antiderivative calculations. I also compared the third-order coefficients with 80-digit mpmath integrals for 1, log(rho) and log(rho)**2, at indices 5, n // 2 and n - 3 for n = 50, 100, 500. The largest relative errors at these selected nodes were 14.5%, 14.7% and 14.8% for degrees 0, 1 and 2; promoting the coefficient calculations reduced them below 8e-10.

A local candidate change promotes the reconstructed ratios before evaluating the second- and third-order coefficients. That reduces the 500-step trajectory error to 3.6e-5 for float64 samples and 8.9e-5 for float32 samples. This is narrower than the metadata-promotion control in the table and leaves other float32 scalar calculations in place. The remaining error has not been fully attributed. All 14 local tests pass with the change; 11 fail before it. The tests also assert that the output dtype stays unchanged. I'd be happy to work on a fix if this approach makes sense.

The default schedule here has distinct points and a positive final ratio, sigmas[-1] = 0.010001329705119133. This DEIS implementation has no final_sigmas_type option. Its lower-order-final fallback applies only below 15 steps, so it is inactive in these runs. Sigma metadata stays on CPU.

This is a scheduler correctness reproduction. I haven't measured generated-image quality, GPU behavior or performance.

Reproduction

import math
import torch
from diffusers import DEISMultistepScheduler


def run(steps, dtype, double_scalars):
    scheduler = DEISMultistepScheduler(solver_order=3)
    scheduler.set_timesteps(steps)
    original_sigmas = scheduler.sigmas.clone()
    if double_scalars:
        scheduler.sigmas = scheduler.sigmas.double()
    assert torch.equal(original_sigmas, scheduler.sigmas.float())

    rho0 = float(scheduler.sigmas[0])
    sample = torch.full(
        (1, 1), rho0 / math.sqrt(1 + rho0**2), dtype=dtype
    )
    constant = sample.item() * math.sqrt(1 + rho0**2) - rho0
    epsilon = torch.ones_like(sample)
    for t in scheduler.timesteps:
        sample = scheduler.step(epsilon, t, sample).prev_sample

    rho = float(scheduler.sigmas[-1])
    expected = (constant + rho) / math.sqrt(1 + rho**2)
    assert sample.dtype == dtype
    print(steps, dtype, double_scalars, sample.item(), expected)


for dtype in (torch.float64, torch.float32):
    for steps in (50, 100, 500):
        run(steps, dtype, False)
        run(steps, dtype, True)

Logs

50 torch.float64 False 0.016792173382122744 0.010000829543158653
50 torch.float64 True 0.010000829540443906 0.010000829543158653
100 torch.float64 False 0.0544108477686773 0.010000829543158653
100 torch.float64 True 0.010000829656155221 0.010000829543158653
500 torch.float64 False 4.476663502336187 0.010000829543158653
500 torch.float64 True 0.010000829163227195 0.010000829543158653
50 torch.float32 False 0.016786660999059677 0.009996639490210355
50 torch.float32 True 0.010010620579123497 0.009996639490210355
100 torch.float32 False 0.05440002307295799 0.009996639490210355
100 torch.float32 True 0.010017460212111473 0.009996639490210355
500 torch.float32 False 4.476640701293945 0.009996639490210355
500 torch.float32 True 0.010039642453193665 0.009996639490210355

System info

diffusers 0.42.0.dev0 at 537887041938fc6fe61ad583c3ae77477c9ebac9, PyTorch 2.13.0+cu130, Python 3.12.13, Linux, CPU. NumPy 2.5.0, huggingface-hub 1.32.0.

Relevant source.

I checked the existing DEIS issues, including the precision discussion in #4167. That discussion concerns differences in generated beta/sigma values; this reproduction keeps those values identical. I did not find a matching coefficient-cancellation report.