[Performance Degredation] 2x NFS Performance Degradation Between 5.15.0 and 6.17.0

[arikrinberg-cloud]

Hi,

We're seeing a large performance degradation between Kernel 5.15 and 6.17. We ran the experiment below on GCP, and found a worsening in 25% of IOPs and 2x latency. This experiment was done without altering the kernel. We did bisect the kernel and found the offending commit was caec5ebe77f97d948dcf46f07d622bda7f1f6dfd.

Server

The server is a c4-highmem-96 machine with 8TB hyperdisk balanced with 160K provisioned IOPS and 2,400 MB/s provisioned throughput

Clients

The clients are c4-highmem-2 machines

knfsd

The server was created with the following export:

arikrinberg@nfs-server:~$ cat /etc/exports
/srv/nfs/shared_data 10.128.0.0/20(rw,sync,no_subtree_check)

No other changes were done.

Test

FIO conf

arikrinberg@vm-ubuntu-2025-c4:~$ cat fio.conf 
[global]
bs=4096
direct=0
buffer_compress_percentage=50
ioengine=io_uring
fallocate=truncate
group_reporting=1
time_based=1
runtime=60
ramp_time=0
name=test-job
file_service_type=random
filesize=2097152
allow_file_create=0
nrfiles=512
directory=/mnt/nfs/dataset
exitall_on_error=1
disk_util=0

[job0.0]
iodepth=100
readwrite=randread
filename_format=$filenum

Ubuntu 2022 (5.15.0)

arikrinberg@vm-ubuntu-2022-c4:~$ fio fio.conf
job0.0: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=100
fio-3.28
Starting 1 process
Jobs: 1 (f=512): [r(1)][100.0%][r=326MiB/s][r=83.5k IOPS][eta 00m:00s]
job0.0: (groupid=0, jobs=1): err= 0: pid=3623: Sun Nov 16 11:07:15 2025
  read: IOPS=82.9k, BW=324MiB/s (339MB/s)(19.0GiB/60001msec)
    slat (nsec): min=1466, max=1578.9k, avg=8946.09, stdev=12993.26
    clat (usec): min=3, max=46716, avg=1369.05, stdev=483.16
     lat (usec): min=4, max=46723, avg=1378.11, stdev=484.89
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[  791], 10.00th=[  898], 20.00th=[ 1020],
     | 30.00th=[ 1123], 40.00th=[ 1205], 50.00th=[ 1319], 60.00th=[ 1434],
     | 70.00th=[ 1565], 80.00th=[ 1713], 90.00th=[ 1909], 95.00th=[ 2057],
     | 99.00th=[ 2376], 99.50th=[ 2606], 99.90th=[ 3458], 99.95th=[ 4293],
     | 99.99th=[10814]
   bw (  KiB/s): min=255680, max=396016, per=100.00%, avg=331705.13, stdev=23227.88, samples=119
   iops        : min=63920, max=99004, avg=82926.24, stdev=5806.97, samples=119
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.46%, 750=3.38%, 1000=14.20%
  lat (msec)   : 2=75.30%, 4=6.60%, 10=0.05%, 20=0.01%, 50=0.01%
  cpu          : usr=6.25%, sys=49.65%, ctx=1596770, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=4971523,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=324MiB/s (339MB/s), 324MiB/s-324MiB/s (339MB/s-339MB/s), io=19.0GiB (20.4GB), run=60001-60001msec

Ubuntu 2025 (6.17.0)

arikrinberg@vm-ubuntu-2025-c4:~$ fio fio.conf                                                                                                                                                                                                                                             
job0.0: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=100                                                                                                                                                                          
fio-3.39                                                                                                                                                                                                                                                                                  
Starting 1 process                                                                                                                                                                                                                                                                        
Jobs: 1 (f=512): [r(1)][100.0%][r=239MiB/s][r=61.3k IOPS][eta 00m:00s]                                                                                                                                                                                                                    
job0.0: (groupid=0, jobs=1): err= 0: pid=2783: Sun Nov 16 11:06:11 2025                                                                                                                                                                                                                   
  read: IOPS=60.4k, BW=236MiB/s (247MB/s)(13.8GiB/60002msec)                                                                                                                                                                                                                              
    slat (nsec): min=311, max=849646, avg=1683.90, stdev=7973.42                                                                                                                                                                                                                          
    clat (usec): min=102, max=31129, avg=1653.31, stdev=266.12                                                                                                                                                                                                                            
     lat (usec): min=107, max=31130, avg=1654.99, stdev=265.85                                                                                                                                                                                                                            
    clat percentiles (usec):                                                                                                                                                                                                                                                              
     |  1.00th=[ 1139],  5.00th=[ 1450], 10.00th=[ 1500], 20.00th=[ 1549],                                                                                                                                                                                                                
     | 30.00th=[ 1582], 40.00th=[ 1598], 50.00th=[ 1631], 60.00th=[ 1663],                                                                                                                                                                                                                
     | 70.00th=[ 1696], 80.00th=[ 1745], 90.00th=[ 1827], 95.00th=[ 1926],                                                                                                                                                                                                                
     | 99.00th=[ 2278], 99.50th=[ 2540], 99.90th=[ 3130], 99.95th=[ 3359],                                                                                                                                                                                                                
     | 99.99th=[ 9503]                                                                                                                                                                                                                                                                    
   bw (  KiB/s): min=192600, max=253600, per=100.00%, avg=241583.00, stdev=7070.59, samples=119                                                                                                                                                                                           
   iops        : min=48150, max=63400, avg=60395.75, stdev=1767.65, samples=119                                                                                                                                                                                                           
  lat (usec)   : 250=0.01%, 500=0.08%, 750=0.18%, 1000=0.36%                                                                                                                                                                                                                              
  lat (msec)   : 2=96.38%, 4=2.96%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=5.61%, sys=16.19%, ctx=2294269, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=3621379,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=100

Run status group 0 (all jobs):
   READ: bw=236MiB/s (247MB/s), 236MiB/s-236MiB/s (247MB/s-247MB/s), io=13.8GiB (14.8GB), run=60002-60002msec

[axboe]

Try and add this patch on the newer kernel:

diff --git a/fs/nfs/file.c b/fs/nfs/file.c
index d020aab40c64..7f54e0be8456 100644
--- a/fs/nfs/file.c
+++ b/fs/nfs/file.c
@@ -73,7 +73,7 @@ nfs_file_open(struct inode *inode, struct file *filp)
 
 	res = nfs_open(inode, filp);
 	if (res == 0)
-		filp->f_mode |= FMODE_CAN_ODIRECT;
+		filp->f_mode |= FMODE_CAN_ODIRECT | FMODE_NOWAIT;
 	return res;
 }

[arikrinberg-cloud]

Just to prefix:

We are measuring Filestore NFS performance, so I was quickly set up to test that.

5.15.0 performance was:

Jobs: 1 (f=512): [r(1)][100.0%][r=64.0MiB/s][r=16.4k IOPS][eta 00m:00s]
job0.0: (groupid=0, jobs=1): err= 0: pid=1644: Tue Nov 11 12:46:16 2025
  read: IOPS=14.8k, BW=57.9MiB/s (60.7MB/s)(3476MiB/60003msec)
    slat (usec): min=3, max=13507, avg=22.47, stdev=56.07
    clat (nsec): min=270, max=151252k, avg=3454120.24, stdev=2305582.73
     lat (usec): min=7, max=151342, avg=3476.88, stdev=2305.92
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[    9], 99.50th=[   11], 99.90th=[   25], 99.95th=[   53],

6.17.0 performance was:

Jobs: 1 (f=512): [r(1)][100.0%][r=13.5MiB/s][r=3458 IOPS][eta 00m:00s]
job0.0: (groupid=0, jobs=1): err= 0: pid=28407: Tue Oct 28 12:22:20 2025
  read: IOPS=3438, BW=13.4MiB/s (14.1MB/s)(2418MiB/180015msec)
    slat (nsec): min=1246, max=1207.2k, avg=3568.25, stdev=8199.75
    clat (usec): min=2609, max=78498, avg=15460.84, stdev=2512.45
     lat (usec): min=2612, max=78608, avg=15464.61, stdev=2511.92
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12256], 10.00th=[12649], 20.00th=[13435],
     | 30.00th=[14091], 40.00th=[14746], 50.00th=[15270], 60.00th=[15926],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[21890], 99.50th=[24773], 99.90th=[35914], 99.95th=[44303],
     | 99.99th=[64750]

With the patch the performance is:

Jobs: 1 (f=512): [r(1)][100.0%][r=38.2MiB/s][r=9784 IOPS][eta 00m:00s]
job0.0: (groupid=0, jobs=1): err= 0: pid=1619: Sun Nov 16 19:02:42 2025
  read: IOPS=9650, BW=37.7MiB/s (39.5MB/s)(2262MiB/60008msec)
    slat (nsec): min=108, max=1091.8k, avg=872.23, stdev=5097.21
    clat (usec): min=1422, max=71711, avg=6557.64, stdev=1731.86
     lat (usec): min=1429, max=71720, avg=6558.52, stdev=1732.00
    clat percentiles (usec):
     |  1.00th=[ 3949],  5.00th=[ 4948], 10.00th=[ 5342], 20.00th=[ 5669],
     | 30.00th=[ 5866], 40.00th=[ 6063], 50.00th=[ 6194], 60.00th=[ 6390],
     | 70.00th=[ 6652], 80.00th=[ 7046], 90.00th=[ 8160], 95.00th=[ 9634],
     | 99.00th=[12649], 99.50th=[14091], 99.90th=[20055], 99.95th=[25560],
     | 99.99th=[46924]

So a super large increase!

The latency is still 4ms, and the IOPs are lower, but its way closer.

I'm wondering if it is due to the move to folios? When we investigated using a tcp dump we saw that the average latency for NFS operations was ~2.5ms once it reached the NIC, and using offcputime-bpfcc we found that many threads were waiting on folio_wait_bit_common.

[axboe]

What version nfs are you using?

[arikrinberg-cloud]

NFS v3.

Also, are you sending a patch to the kernel with FMODE_NOWAIT or should I?

[arikrinberg-cloud]

Hi @axboe, just wanted to ask regarding the patch to the kernel?