Discrepancy in FID and LPIPS Scores During Reproduction Attempt

[Dingyi-Yao]

Description

Hi! Thank you for your work!
I’m a beginner trying to reproduce results from the paper. After generating images using the provided code, I computed FID and LPIPS metrics([https://github.com//issues/3]) but observed significant differences compared to Table 1 in the paper (FID: 274.52 vs. reported 55.30, LPIPS: 0.494 vs. reported 0.36).

File Structure

evaluate_img/
├── exp1_200/
│   ├── CACTIF_transfer_1.png
│   ├── CACTIF_transfer_2.png
│   └── ...
├── origin_200/
│   ├── 00001.png
│   ├── 00002.png
│   └── ...
└── citystyle_1/
    ├── 1.png
    ├── 2.png
    └── ...

Steps to Reproduce

1. Image Generation & Structure:

   • Generated 200 output images via CACTIF transfer, stored in evaluate_img/exp1_200/ (nb_img_per_style: int = 200)
   • Used 200 original GTA images from evaluate_img/origin_200/ .
   • Created style reference by duplicating zurich_000087_000019_leftImg8bit 200 times into evaluate_img/citystyle_1/ .

2. FID Calculation (via pytorch_fid):

   python -m pytorch_fid evaluate_img/citystyle_1 evaluate_img/exp1_200 --device cuda:1  

   Result: FID = 274.52 (expected: 55.30).

3. LPIPS Calculation (modified script):
   Used a custom lpips_sim2real.py to handle mismatched filenames:

   python PerceptualSimilarity/lpips_sim2real.py -d0 evaluate_img/origin_200 -d1 evaluate_img/exp1_200 -o evaluate_img/example_dists.txt --use_gpu --gpu_id 1  

   Result: Avg. LPIPS = 0.494 (expected: 0.36).

Request for Help

As a beginner working to reproduce these results, I would be deeply grateful for any guidance on where I might be going wrong. Any overlooked steps in the reproduction process that might explain the metric discrepancies?
Thank you for your time and consideration.

[Dingyi-Yao]

And this is my custom lpips_sim2real.py

import argparse
import os
import lpips
import torch
import torch.nn.functional as F
import re

def transfer_number_key(filename):
    match = re.search(r'transfer_(\d+)', filename)
    if match:
        return int(match.group(1))
    return float('inf')  # Place at end if no match found


parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('-d0','--dir0', type=str, default='./imgs/ex_dir0')
parser.add_argument('-d1','--dir1', type=str, default='./imgs/ex_dir1')
parser.add_argument('-o','--out', type=str, default='./imgs/example_dists.txt')
parser.add_argument('-v','--version', type=str, default='0.1')
parser.add_argument('--use_gpu', action='store_true', help='Enable GPU acceleration')
parser.add_argument('--gpu_id', type=int, default=0, help='Specify GPU device ID')

opt = parser.parse_args()

## Configure GPU
if(opt.use_gpu):
    if torch.cuda.is_available():
        torch.cuda.set_device(opt.gpu_id)
        print(f"Using GPU: {opt.gpu_id}")
    else:
        print("Warning: CUDA device not detected, falling back to CPU")
        opt.use_gpu = False

## Initialize model
loss_fn = lpips.LPIPS(net='alex',version=opt.version)
if(opt.use_gpu):
    loss_fn.cuda()

# Process directories
f = open(opt.out,'w')
# Get sorted file lists from both directories
files0 = sorted(os.listdir(opt.dir0))
files1 = sorted(os.listdir(opt.dir1), key=transfer_number_key)

# Verify equal number of files in both directories
min_files = min(len(files0), len(files1))
if len(files0) != len(files1):
    print(f"Warning: Unequal file count. Dir0: {len(files0)} files, Dir1: {len(files1)} files.")
    print(f"Processing first {min_files} file pairs only.")
    
total_dist = 0
# Process file pairs
for i in range(min_files):
    file0 = files0[i]
    file1 = files1[i]
    
    # Load images
    img0 = lpips.im2tensor(lpips.load_image(os.path.join(opt.dir0, file0)))
    img1 = lpips.im2tensor(lpips.load_image(os.path.join(opt.dir1, file1)))
    
    # Resize img0 to match img1 dimensions
    img0 = F.interpolate(img0, size=(img1.shape[2], img1.shape[3]), mode='bilinear', align_corners=False)

    if opt.use_gpu:
        img0 = img0.cuda()
        img1 = img1.cuda()
    
    # Compute LPIPS distance
    dist01 = loss_fn.forward(img0, img1).item()
    total_dist += dist01
    print(f"{file0} vs {file1}: {dist01:.3f}")

# Calculate average distance
if min_files > 0:
    avg_dist = total_dist / min_files
    print(f"Average distance: {avg_dist:.6f}")
    f.writelines(f"Average distance: {avg_dist:.6f}\n")

f.close()