Add initial end-to-end CUDA FGMRES solver path - #2825
Merged
Conversation
LwhJesse
marked this pull request as ready for review
June 3, 2026 05:20
pcarruscag
reviewed
Jun 13, 2026
LwhJesse
force-pushed
the
gpu/initial-cuda-fgmres
branch
from
June 17, 2026 14:24
8982fcb to
d821ca0
Compare
LwhJesse
force-pushed
the
gpu/initial-cuda-fgmres
branch
from
June 18, 2026 05:59
592b302 to
5b01f21
Compare
Move the FGMRES iteration into one shared implementation and select host or CUDA vector-operation backends from the existing solver entry point. Keep the CUDA path GPU-resident for SpMV, Jacobi, dot/norm, and vector updates, with only scalar reductions and final solution synchronization crossing back to the host.
LwhJesse
force-pushed
the
gpu/initial-cuda-fgmres
branch
from
June 18, 2026 11:37
5b01f21 to
a875f56
Compare
pcarruscag
reviewed
Jun 21, 2026
ThreadSanitizer hybrid_regression jobs reported a data race in CSysVector::MarkHostDataModified() on host/device validity flags. Avoid updating those CUDA-only validity flags in non-CUDA builds.
12 tasks
# Conflicts: # Common/include/linear_algebra/CSysMatrix.hpp # Common/src/linear_algebra/CSysMatrix.cpp # Common/src/linear_algebra/CSysMatrixGPU.cu # meson.build
- meson.build: fix CUDA arch for Ada Lovelace GPUs, use find_library for cudart (Ubuntu's distro-packaged CUDA toolkit installs libs outside the path meson's cuda dependency() module searches), skip the AMX-tile intrinsics header nvcc's frontend can't parse, and point nvcc's host compiler at the actual C++ compiler so MPI link flags aren't dropped when nvcc becomes the final linker for CUDA-containing targets. - Common/src/meson.build: stop forwarding CODI_REVERSE_TYPE/ CODI_FORWARD_TYPE into CUDA compilation; the GPU kernels only ever operate on plain floating-point types and nvcc's device-code frontend cannot parse CoDiPack's tape/event-system machinery. - CSysMatrixGPU.cu: fix BlockLDU_SpMV_kernel's row/col index pointer types to match the LDU struct's actual su2uint (uint32_t) storage. - CSysVector/CSysMatrix/CMatrixVectorProduct: the GPU dispatch paths (GPUDot, HtDTransfer, AssignDeviceExpression, GPUMatrixVectorProduct, Jacobi preconditioner GPU hooks) were only gated at runtime, but are only ever instantiated for non-AD scalar types. Add a su2_gpu_capable_v compile-time trait and gate every call site with if constexpr so AD-typed instantiations never reference those symbols and correctly fall back to host computation.
pcarruscag
approved these changes
Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Changes
This PR adds an end-to-end CUDA linear solve path: the Krylov solvers keep their host control flow, and
CSysVectoroperations, the SpMV and the Jacobi preconditioner are dispatched to CUDA kernels whenENABLE_CUDA=YES.Transfers are explicit and owned by the object responsible for the data, with no coherency or dirty-flag tracking in
CSysVector/CSysMatrix:CSysMatrixVectorProductuploads the matrix on constructionBuild()CSysSolveuploadsbandxand downloadsxinHandleTemporariesIn/Out, which is also where device evaluation is switched on and offA solve is therefore fully device resident for Identity and Jacobi preconditioners: 2 uploads and 1 download per linear system, independent of the Krylov subspace size.
Implementation notes:
cuBLASfordot/norm, custom kernels for the block-LDU SpMV, the Jacobi apply, andCSysVectorexpression assignmentCSysVectoroperands are captured in expressions by value, so an arbitrary expression tree is trivially copyable into the assignment kernel; the required expression shapes are explicitly instantiated inCSysVectorGPU.cu, consistent with howCSysMatrixis instantiatedLINEAR_SOLVER_PREC= NONE(identity)Related Work
Follows the review direction in #2822 (show a working end-to-end GPU linear solve before splitting out infrastructure) and the implementation preferences in #2816.
Validation
JACOBI, FGMRES +NONE, FGMRES +ILU(host preconditioner path) and BCGSTAB. Results agree to the printed precision in double, and to ~6 significant figures in mixed precision. BCGSTAB agrees exactly once the linear system is converged (LINEAR_SOLVER_ERROR=1e-10); at loose tolerances the two paths diverge through BCGSTAB's own sensitivity, not a difference in the algebra.OMP_NUM_THREADS=1and4give bit-identical results on the GPU path.enable-cuda+with-omp, mixed / normal / single precision.NONE, plus one download/upload pair per preconditioner application for ILU.Earlier validation of the original design (6 representative cases,
nsys/ncuprofiling) predates the rework of the transfer and dispatch model and should be repeated.PR Checklist
pre-commit run --allto format old commits.