perf(mpi): token protocol for the neighbour mesh and graduation exchanges - #477
Draft
gouarin wants to merge 1 commit into
Draft
perf(mpi): token protocol for the neighbour mesh and graduation exchanges#477gouarin wants to merge 1 commit into
gouarin wants to merge 1 commit into
Conversation
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 15 |
| Duplication | -5 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
5 tasks
gouarin
force-pushed
the
mpi-comm-reduction
branch
3 times, most recently
from
July 22, 2026 18:22
4aae14b to
14877b2
Compare
…nges Every mesh construction serialized and sent the full mesh state to each neighbour three times (subdomain, cells, then the whole mesh with every derived mesh id), and every iteration of the graduation fixed point re-sent the full cell array - even when the sender's data had not changed since the previous exchange. Replace these exchanges by a token protocol: a one-int header announces whether the serialized payload follows; when it does not, the receiver keeps its cached copy of the neighbour's data. Senders only skip the payload when the receivers provably hold identical data: - mesh exchanges: cells geometrically identical to the reference mesh (the CellArray equality used for the adaptation fixed point), same neighbour set, and - for the whole-mesh exchange, whose derived ids depend on the neighbours' cells - a token received from every neighbour in the cells exchange. find_neighbourhood now carries the previously exchanged neighbour meshes over to the rebuilt neighbourhood, which is the receive-side cache. - graduation: the existing ca_changed flag is exactly "modified since the last exchange"; the neighbour cell arrays persist across the fixed-point iterations, so a rank at its local fixed point stops resending its identical cell array while a neighbour keeps refining. advection_2d (Tf 0.1, Apple M, A/B interleaved, vs mesh-update-fastpath): - 8 ranks, libfabric sockets provider: 167 -> 120 s (-28%) - 8 ranks, FI_PROVIDER=tcp: -3%; 4 ranks: -2%; serial: parity The remaining MPI cost on a sane transport is the NUMBER of synchronous exchange rounds (collectives and per-iteration exchanges), not the payload volume - reducing rounds is the next lever. Outputs are bit-identical by construction (h5diff clean in serial and at 4 and 8 ranks); tests 350/350 serial, 24/24 MPI (2/3/4 ranks).
gouarin
force-pushed
the
mpi-comm-reduction
branch
from
July 22, 2026 18:29
14877b2 to
c8be3ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Available tags: 'build', 'chore', 'ci', 'docs', 'feat', 'fix', 'perf', 'refactor', 'revert', 'style', 'test'
Description
Every mesh construction serialized and sent the full mesh state to each neighbour three times (subdomain, cells, then the whole mesh with every derived mesh id), and every iteration of the graduation fixed point re-sent the full cell array - even when the sender's data had not changed since the previous exchange.
This PR replaces those exchanges by a token protocol: a one-int header announces whether the serialized payload follows; when it does not, the receiver keeps its cached copy of the neighbour's data. A sender only skips the payload when the receivers provably hold identical data:
update_neighbour_subdomain,update_meshid_neighbour,update_mesh_neighbour): cells geometrically identical to the reference mesh (same_cells, from perf(mesh): detect the adaptation fixed point before constructing the new mesh #476), same neighbour set and - for the whole-mesh exchange, whose derived ids depend on the neighbours' cells - a token received from every neighbour in the cells exchange.find_neighbourhoodnow carries the previously exchanged neighbour meshes over to the rebuilt neighbourhood; they are the receive-side cache.ca_changedflag is exactly "modified since the last exchange"; the neighbours' cell arrays now persist across the fixed-point iterations, so a rank at its local fixed point stops resending its identical cell array while a neighbour keeps refining.The exchanges stay collective over the (symmetric) neighbourhood, so the communication pattern remains matched.
Measurements on
advection_2d(Tf 0.1, Apple M, MPICH + libfabric, interleaved A/B vs #476):socketsproviderFI_PROVIDER=tcpTokens cut the payload volume, which pays hugely on expensive transports and grows with rank count and mesh size. On a sane transport at low rank counts, the remaining MPI cost of the adaptation loop is the number of synchronous exchange rounds - addressed in #478.
Related issue
None.
How has this been tested?
demos/FiniteVolume/advection_2d.cpp, Tf 0.5);tests/: 350/350 serial;tests/mpi/: 24/24 at 2/3/4 ranks, including the load-balancing migration tests which exercise the carried-over neighbour caches;Code of Conduct
By submitting this PR, you agree to follow our Code of Conduct