Replication error.

[zhuyu-cs]

Since there is no test script for FID in current code, I was trying to reproduce the result of cifar10 in paper Table1 based on clean-fid. I have changed the config to use denoiser during sampling, but the final FID is not 2.72, but 12.72, and IS=10.33.

So how can I implement a proper FID test script to reproduce the results in the text?

Thanks!

[louaaron]

The FID script is the one from https://github.com/yang-song/score_sde_pytorch.

The results seem flipped (the denoiser should hurt the IS but help the FID), so it may be possible that there is a bug there.

[zhuyu-cs]

Hi, sorry for the late reply. I have retest the FID follow https://github.com/yang-song/score_sde_pytorch. The results are as follows:
Predictor-Denoiser: IS: 10.42853, FID: 13.12402,
Predictor: IS: 10.36311, FID: 13.20688

I have no idea where the problem led to this results. Any help would be appreciated!

[saeidnp]

Hi there,

I am also observing the same issue. The FIDs values are around 12-13, both for the provided checkpoint and re-trained models. The values I am getting for the provided checkpoint are IS: 10.28136 and FID: 12.83104.

I am using (torch-fidelity)[https://github.com/toshas/torch-fidelity]. It has been tested to provide results consistent with the original FID implementation. I have also compared it with different implementations of FID to verify its accuracy. As such, I believe the FID computation code is not an issue here.