Skip to content

Fixed per-thread array bounds on all GPU compilers, with a generated state-vector layout #2081

Open

Description

@sbryngelson

Summary

Per-thread arrays in GPU kernels now get compile-time extents in every GPU simulation build, from one place, and case optimization supplies exact extents including sys_size.

Runtime-sized private arrays (dimension(num_fluids), dimension(sys_size)) cannot live in registers and spill to scratch. The USING_AMD guards fixed this for amdflang only; profiling on AFAR 24.3 (rocprofv3) showed WENO 11x and HLLC 4.9x slower without them. #1938 turned the same fixed bounds on for every compiler, and CI showed CCE 12-36% and NVHPC OpenMP 9-62% faster (AMD unchanged), but GNU CPU up to 39% slower. So this PR applies them to GPU builds only.

Changes (four commits)

  1. BOUND() replaces the USING_AMD guards. The 83 copy-pasted #:if [not MFC_CASE_OPTIMIZATION and] USING_AMD blocks become single declarations such as dimension(${BOUND('num_fluids')}$). BOUND returns the fixed maximum from one table (num_fluids 3, nb 3, sys_size 70 or 10 + species, WENO and QBMM extents, and so on) or the runtime extent. A converter checked every existing literal against that table. MFC_FIXED_BOUNDS (CMake, default ON) is decided per target: only the offloaded simulation uses fixed bounds, so pre_process/post_process callers always match the routines they call. s_check_amd becomes s_check_fixed_bounds, and the sys_size limit is checked right after the layout is computed (the old input-time check ran before sys_size existed).
  2. The state-vector layout is generated from one table. toolchain/mfc/params/eqn_layout.py lists each model's fields with their size and condition as small Python expressions. The params codegen turns them into generated_eqn_idx.fpp, replacing about 140 hand-written lines of s_initialize_eqn_idx. Checked against master's routine over all 7,962,624 combinations of model, the 13 layout flags, 1D/multi-D, and the counts: every eqn_idx field and sys_size matches.
  3. Case optimization bakes in exact extents. It evaluates the same table for the case and writes CASE_OPT_SIZES (sys_size and the hypoelastic stress count) into case.fpp, so BOUND gives exact compile-time extents under case optimization on every compiler. The sys_size variable stays a runtime variable, so nothing that declares or updates it on the device changes. Simulation checks at startup that the runtime layout matches the baked-in values. Since CASE_OPT_SIZES is in case.fpp, it is part of the build fingerprint.
  4. Fixed bounds are on for every GPU compiler (NVHPC and CCE, OpenACC and OpenMP); CPU builds keep runtime extents. Docs describe BOUND().

Behavior change

GPU simulation builds without --case-optimization now accept at most 3 fluids, nb <= 3, and sys_size <= 70 (or 10 + species) on every GPU compiler, as amdflang already did; larger cases rebuild with --case-optimization, and the error message says so. CPU builds and pre_process/post_process are unaffected.

Testing

  • Generated code: for amdflang simulation, and for gfortran simulation/pre_process/post_process, the preprocessed Fortran matches master apart from the checker rename and messages.
  • HPCFund MI210, AFAR 24.3, --gpu mp, release: full test suite, 718/718 passed.
  • gfortran CPU: full test suite, 718/718 passed.
  • Case-optimized 5eq_rk3_weno3_hllc on GPU: runs with CASE_OPT_SIZES = {'sys_size': 8} baked in (3D, two fluids), HLLC and CBC arrays sized exactly 8, startup check passes; grind 1.93 against 2.50 without case optimization (--mem 2).
  • AMD benchmark against master, same node (--mem 2): geometric-mean ratio 1.000, every case within ±2%. AMD already had fixed bounds, so no change is expected; the gains are on NVHPC and CCE.
  • NVHPC and CCE: the CI Test and Bench jobs on this PR.

Status: a fix/implementation for this is already written, in a pull request that has been closed for queue management.

It was merge-clean and had been brought up to date with master, so resuming should be straightforward.

Tracked with pr-to-do. Filed with Claude Code.

Activity

  1. added
    pr-to-doFix already written in a closed PR; reopen when review capacity allows
    on Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pr-to-doFix already written in a closed PR; reopen when review capacity allows

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions