Repository navigation
Fixed per-thread array bounds on all GPU compilers, with a generated state-vector layout #2081
Description
Summary
Per-thread arrays in GPU kernels now get compile-time extents in every GPU simulation build, from one place, and case optimization supplies exact extents including sys_size.
Runtime-sized private arrays (dimension(num_fluids), dimension(sys_size)) cannot live in registers and spill to scratch. The USING_AMD guards fixed this for amdflang only; profiling on AFAR 24.3 (rocprofv3) showed WENO 11x and HLLC 4.9x slower without them. #1938 turned the same fixed bounds on for every compiler, and CI showed CCE 12-36% and NVHPC OpenMP 9-62% faster (AMD unchanged), but GNU CPU up to 39% slower. So this PR applies them to GPU builds only.
Changes (four commits)
BOUND()replaces theUSING_AMDguards. The 83 copy-pasted#:if [not MFC_CASE_OPTIMIZATION and] USING_AMDblocks become single declarations such asdimension(${BOUND('num_fluids')}$).BOUNDreturns the fixed maximum from one table (num_fluids3,nb3,sys_size70 or 10 + species, WENO and QBMM extents, and so on) or the runtime extent. A converter checked every existing literal against that table.MFC_FIXED_BOUNDS(CMake, default ON) is decided per target: only the offloadedsimulationuses fixed bounds, sopre_process/post_processcallers always match the routines they call.s_check_amdbecomess_check_fixed_bounds, and thesys_sizelimit is checked right after the layout is computed (the old input-time check ran beforesys_sizeexisted).- The state-vector layout is generated from one table.
toolchain/mfc/params/eqn_layout.pylists each model's fields with their size and condition as small Python expressions. The params codegen turns them intogenerated_eqn_idx.fpp, replacing about 140 hand-written lines ofs_initialize_eqn_idx. Checked against master's routine over all 7,962,624 combinations of model, the 13 layout flags, 1D/multi-D, and the counts: everyeqn_idxfield andsys_sizematches. - Case optimization bakes in exact extents. It evaluates the same table for the case and writes
CASE_OPT_SIZES(sys_sizeand the hypoelastic stress count) intocase.fpp, soBOUNDgives exact compile-time extents under case optimization on every compiler. Thesys_sizevariable stays a runtime variable, so nothing that declares or updates it on the device changes. Simulation checks at startup that the runtime layout matches the baked-in values. SinceCASE_OPT_SIZESis incase.fpp, it is part of the build fingerprint. - Fixed bounds are on for every GPU compiler (NVHPC and CCE, OpenACC and OpenMP); CPU builds keep runtime extents. Docs describe
BOUND().
Behavior change
GPU simulation builds without --case-optimization now accept at most 3 fluids, nb <= 3, and sys_size <= 70 (or 10 + species) on every GPU compiler, as amdflang already did; larger cases rebuild with --case-optimization, and the error message says so. CPU builds and pre_process/post_process are unaffected.
Testing
- Generated code: for amdflang
simulation, and for gfortransimulation/pre_process/post_process, the preprocessed Fortran matches master apart from the checker rename and messages. - HPCFund MI210, AFAR 24.3,
--gpu mp, release: full test suite, 718/718 passed. - gfortran CPU: full test suite, 718/718 passed.
- Case-optimized
5eq_rk3_weno3_hllcon GPU: runs withCASE_OPT_SIZES = {'sys_size': 8}baked in (3D, two fluids), HLLC and CBC arrays sized exactly 8, startup check passes; grind 1.93 against 2.50 without case optimization (--mem 2). - AMD benchmark against master, same node (
--mem 2): geometric-mean ratio 1.000, every case within ±2%. AMD already had fixed bounds, so no change is expected; the gains are on NVHPC and CCE. - NVHPC and CCE: the CI Test and Bench jobs on this PR.
Status: a fix/implementation for this is already written, in a pull request that has been closed for queue management.
- Closed PR: Fixed per-thread array bounds on all GPU compilers, with a generated state-vector layout #1941
- Branch:
fixed-boundson thesbryngelsonfork, at2dcb6ed3c1(branch retained, not deleted) - Size: +954/-1226 across 39 files, 13 commits
- Where it sat: MERGEABLE; CI 19 checks queued (master was merged in today, so CI had not reported yet)
- Reopen with:
gh pr reopen 1941 -R MFlowCode/MFC
It was merge-clean and had been brought up to date with master, so resuming should be straightforward.
Tracked with pr-to-do. Filed with Claude Code.
Activity