Skip to content

SCC26: Use nonblocking MPI for halo exchange - #2090

Draft
RedTimesZero wants to merge 1 commit into
MFlowCode:masterfrom
RedTimesZero:feature/halo-nonblocking
Draft

RedTimesZero wants to merge 1 commit into
MFlowCode:masterfrom
RedTimesZero:feature/halo-nonblocking

Conversation

@RedTimesZero

@RedTimesZero RedTimesZero commented Oct 17, 2026 •

Copy link
Copy Markdown

Description

This PR is from the NTHU team ( Team NTHU SCC ) participating in the SC26 Student Cluster Competition.
Replace the two blocking MPI_SENDRECV calls per direction in the halo exchange with one MPI_Isend/MPI_Irecv + MPI_Waitall, so a rank no longer waits for the first neighbour before starting the second side.

Fixes #2089

Changes

  • s_mpi_sendrecv_variables_buffers is split into s_mpi_pack_halo_buffer, s_mpi_exchange_halo_buffers and s_mpi_unpack_halo_buffer, around a small halo_side descriptor (peer, tags, pack/unpack offsets, buffer offset).
  • buff_send/buff_recv are split into two halves (side 2 starts at halo_size + 1). halo_size now uses integer arithmetic so the halves cannot overlap, and an @:ASSERT checks buffer_count <= halo_size + 1.
  • Tags, packing/unpacking order and GPU_UPDATE staging are unchanged (now per half).
  • m_boundary_common.fpp: comment only, documenting the call order (-1 first, then +1, same arguments).

Documentation: the "How to Extend MPI Halo Exchange" section of docs/documentation/contributing.md still describes the MPI_SENDRECV structure. I left it untouched in this PR; if the approach is accepted I am happy to update it, with whatever wording you prefer.

Testing

Check Result
./mfc.sh test (full, 1 H200, 2 ranks) 725 passed, 0 failed, 39 skipped
Restart file after last step vs master, 2D shock droplet, 1/2/3/4 ranks byte-identical
Same, 3D shock droplet, 6 ranks byte-identical
./mfc.sh format clean
gfortran 13.2 + Open MPI 5.0, CPU, simulation target compiles; not run
Cray ftn, Intel ifx, AMD flang, OpenMP offload not tested

Note: the test suite only exercises 2 ranks, so the pairwise path (two distinct neighbours in one direction) is covered by the shock-droplet comparisons above, not by ./mfc.sh test.

Performance

Grind time (ns/gp/eq/rhs), master -> this PR. Unmodified examples, one rank per H200 (one rank per node), --case-optimization. Mean of 3 runs per variant (2 ranks: 8 runs from two batches).

Case Ranks master this PR Change
2D_shockdroplet (1200x299, 14111 steps) 1 0.4381 0.4345 -0.8%
2D_shockdroplet 2 0.3170 0.3149 -0.7%
2D_shockdroplet 3 0.3265 0.2979 -8.8%
2D_shockdroplet 4 0.2875 0.2611 -9.2%
3D_shockdroplet (201x101x101) 6 0.1668 0.1461 -12.4%

Notes:

  • Why 3D: the 2D case only exercises the x and y directions. The 3D case also exercises the z direction and the 3D branch of the halo_size computation, and adds a 6-rank configuration.
  • 1-2 ranks: there is no second distinct neighbour, so no gain is expected. The differences are within run-to-run variation (at 2 ranks the master and PR ranges overlap).
  • Absolute values: the examples are used unmodified, so the 2D grid is small and absolute grind times are higher than on larger grids. Compare the relative change, not the absolute values.

Contribution Policy

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

@github-actions

Copy link
Copy Markdown

Lines of Code

File Lines Diff
src/common/m_mpi_common.fpp 1567 +73
Directory Lines Diff
common 10599 +73
total 47658 +73

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Claude Code Review

Head SHA: fb5c8d9

Files changed:

  • 2
  • src/common/m_boundary_common.fpp
  • src/common/m_mpi_common.fpp

Findings:
No issues found. Checked in step 4: src/common/m_boundary_common.fpp (context file) and the repo copy of src/common/m_mpi_common.fpp (no prepared context file, so I read it in the working tree). In the paired path, the send and receive tags and the source and destination ranks match across both sides. Each side uses its own half of buff_send and buff_recv, and the allocation of 2*halo_size+2 covers both halves. halo_size is integer(8), so the integer rounding matches the old nint() behavior. Non-RDMA host and device updates cover the right sub-ranges.

@RedTimesZero RedTimesZero changed the title Use nonblocking MPI for halo exchange SCC26: Use nonblocking MPI for halo exchange Oct 17, 2026
Post both sides of a direction with MPI_Isend/MPI_Irecv and wait once
with MPI_Waitall instead of two back-to-back MPI_SENDRECV calls.
halo_size now uses integer arithmetic and is asserted against the
per-direction buffer count.

Fixes MFlowCode#2089

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@RedTimesZero
RedTimesZero force-pushed the feature/halo-nonblocking branch from 8b30b03 to fb5c8d9 Compare October 17, 2026 19:32
@RedTimesZero
RedTimesZero marked this pull request as ready for review October 18, 2026 02:30
@RedTimesZero
RedTimesZero requested a review from a team as a code owner October 18, 2026 02:30
@RedTimesZero
RedTimesZero requested review from sbryngelson and removed request for a team October 18, 2026 02:30
@codecov

codecov Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.86957% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.92%. Comparing base (47568a0) to head (fb5c8d9).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
src/common/m_mpi_common.fpp 60.86% 25 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2090      +/-   ##
==========================================
+ Coverage   61.90%   61.92%   +0.02%     
==========================================
  Files          86       86              
  Lines       22885    22913      +28     
  Branches     3369     3372       +3     
==========================================
+ Hits        14166    14190      +24     
- Misses       6220     6222       +2     
- Partials     2499     2501       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wilfonba
wilfonba marked this pull request as draft October 18, 2026 15:04
@wilfonba

Copy link
Copy Markdown
Collaborator

@RedTimesZero A maintainer will mark your pull request as ready for review when it is suitable to do so

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Non-blocking halo exchange: post both sides of a direction at once

2 participants