OSU tests fail on nvidia/grace

[laraPPr]

[     FAIL ] (15/29) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_node %device_type=cpu /4da98b87 @BotBuildTests:aarch64-nvidia-grace+default
[     FAIL ] (19/29) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_node %device_type=cpu /7a335f5f @BotBuildTests:aarch64-nvidia-grace+default
[     FAIL ] (23/29) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_node /7990dc5c @BotBuildTests:aarch64-nvidia-grace+default
[     FAIL ] (27/29) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_node /f0f45f61 @BotBuildTests:aarch64-nvidia-grace+default

[laraPPr]

@trz42 can you look into this is someone else with access to the bot account at jsc?

[boegel]

from test.outerr.1hfE in /p/project1/ceasybuilders/eessibot/trash_bin/EESSI/software-layer/2026.06.10/2026.06/pr_1515/event_e2a67a90-63ce-11f1-810c-2be1faa23355/run_001/aarch64/nvidia/grace/eessi.io-2025.06-software:

--- rfm_job.out ------ rfm_job.err (last 10 lines) ---
Lmod Warning: Environment variable EESSI_CUDA_DRIVER_VERSION not found. Cannot
ensure that driver version is new enough for CUDA toolkit version: '12.9.1'.
This module will still be loaded, but may not function as expected. Export
EESSI_CUDA_DRIVER_VERSION_SUPPRESS_WARNING=1
While processing the following module(s):
    Module fullname                                     Module Filename
    ---------------                                     ---------------
    OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1  /cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/modules/all/OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1.lua

sh: line 1: srun: command not found
--- rfm_job.err ---
--------------------------------------------------------------------------------
FAILURE INFO for EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_node (run: 1/1)
  * Description:
  * System partition: BotBuildTests
  * Environment: default
  * Test file: /eessi_bot_job/EESSI-test-suite/eessi/testsuite/tests/apps/osu.py
  * Stage directory: /eessi_bot_job/reframe_runs/stage/BotBuildTests/aarch64-nvidia-grace/default/EESSI_OSU_pt2pt_CPU_f0f45f61
  * Node list: jrc0900.jureca
  * Job type: local (id=2077)
  * Dependencies (conceptual): []
  * Dependencies (actual): []
  * Maintainers: []
  * Failing phase: sanity
  * Rerun with '-n /f0f45f61 -p default --system BotBuildTests -r'
  * Reason: sanity error: pattern '^4194304.*Pass' not found in 'rfm_job.out'
--- rfm_job.out (last 10 lines) ---
EESSI_CVMFS_REPO: /cvmfs/software.eessi.io
EESSI_SOFTWARE_SUBDIR: aarch64/nvidia/grace
FULL_MODULEPATH: /cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/modules/all/OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1.lua
--- rfm_job.out ------ rfm_job.err (last 10 lines) ---
 7  osu_bw() [0x4073c0]
 8  osu_bw() [0x403dd4]
 9  /cvmfs/software.eessi.io/versions/2025.06/compat/linux/aarch64/lib/../lib64/libc.so.6(+0x22074) [0x400047b72074]
10  /cvmfs/software.eessi.io/versions/2025.06/compat/linux/aarch64/lib/../lib64/libc.so.6(__libc_start_main+0x98) [0x400047b72148]
11  osu_bw() [0x405170]
=================================
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 2189 on node jrc0900 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
--- rfm_job.err ---

this is interesting:

sh: line 1: srun: command not found

but real problem is probably:

prterun noticed that process rank 1 with PID 2189 on node jrc0900 exited on
signal 11 (Segmentation fault).