Skip to content

perf(mpi): token protocol for the neighbour mesh and graduation exchanges - #477

Draft
gouarin wants to merge 1 commit into
hpc-maths:mainfrom
gouarin:mpi-comm-reduction
Draft

perf(mpi): token protocol for the neighbour mesh and graduation exchanges#477
gouarin wants to merge 1 commit into
hpc-maths:mainfrom
gouarin:mpi-comm-reduction

Conversation

@gouarin

@gouarin gouarin commented Jul 22, 2026

Copy link
Copy Markdown
Contributor
  • I have installed pre-commit locally and use it to validate my commits.
  • The PR title follows the conventional commits convention.
    Available tags: 'build', 'chore', 'ci', 'docs', 'feat', 'fix', 'perf', 'refactor', 'revert', 'style', 'test'
  • This new PR is documented.
  • This new PR is tested.

Description

Rebased on main after #476 was merged.

Every mesh construction serialized and sent the full mesh state to each neighbour three times (subdomain, cells, then the whole mesh with every derived mesh id), and every iteration of the graduation fixed point re-sent the full cell array - even when the sender's data had not changed since the previous exchange.

This PR replaces those exchanges by a token protocol: a one-int header announces whether the serialized payload follows; when it does not, the receiver keeps its cached copy of the neighbour's data. A sender only skips the payload when the receivers provably hold identical data:

  • mesh exchanges (update_neighbour_subdomain, update_meshid_neighbour, update_mesh_neighbour): cells geometrically identical to the reference mesh (same_cells, from perf(mesh): detect the adaptation fixed point before constructing the new mesh #476), same neighbour set and - for the whole-mesh exchange, whose derived ids depend on the neighbours' cells - a token received from every neighbour in the cells exchange. find_neighbourhood now carries the previously exchanged neighbour meshes over to the rebuilt neighbourhood; they are the receive-side cache.
  • graduation: the existing ca_changed flag is exactly "modified since the last exchange"; the neighbours' cell arrays now persist across the fixed-point iterations, so a rank at its local fixed point stops resending its identical cell array while a neighbour keeps refining.

The exchanges stay collective over the (symmetric) neighbourhood, so the communication pattern remains matched.

Measurements on advection_2d (Tf 0.1, Apple M, MPICH + libfabric, interleaved A/B vs #476):

Config before after
8 ranks, libfabric sockets provider 167 s 120 s (-28%)
8 ranks, FI_PROVIDER=tcp 7.6-7.7 s 7.4-7.5 s (-3%)
4 ranks 3.09-3.14 s 3.03-3.10 s (-2%)
serial parity parity

Tokens cut the payload volume, which pays hugely on expensive transports and grows with rank count and mesh size. On a sane transport at low rank counts, the remaining MPI cost of the adaptation loop is the number of synchronous exchange rounds - addressed in #478.

Related issue

None.

How has this been tested?

Code of Conduct

By submitting this PR, you agree to follow our Code of Conduct

  • I agree to follow this project's Code of Conduct

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 15 complexity · -5 duplication

Metric Results
Complexity 15
Duplication -5

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

…nges

Every mesh construction serialized and sent the full mesh state to each
neighbour three times (subdomain, cells, then the whole mesh with every
derived mesh id), and every iteration of the graduation fixed point
re-sent the full cell array - even when the sender's data had not changed
since the previous exchange.

Replace these exchanges by a token protocol: a one-int header announces
whether the serialized payload follows; when it does not, the receiver
keeps its cached copy of the neighbour's data. Senders only skip the
payload when the receivers provably hold identical data:

- mesh exchanges: cells geometrically identical to the reference mesh
  (the CellArray equality used for the adaptation fixed point), same
  neighbour set, and - for the whole-mesh exchange, whose derived ids
  depend on the neighbours' cells - a token received from every neighbour
  in the cells exchange. find_neighbourhood now carries the previously
  exchanged neighbour meshes over to the rebuilt neighbourhood, which is
  the receive-side cache.
- graduation: the existing ca_changed flag is exactly "modified since the
  last exchange"; the neighbour cell arrays persist across the fixed-point
  iterations, so a rank at its local fixed point stops resending its
  identical cell array while a neighbour keeps refining.

advection_2d (Tf 0.1, Apple M, A/B interleaved, vs mesh-update-fastpath):
- 8 ranks, libfabric sockets provider: 167 -> 120 s (-28%)
- 8 ranks, FI_PROVIDER=tcp: -3%; 4 ranks: -2%; serial: parity
The remaining MPI cost on a sane transport is the NUMBER of synchronous
exchange rounds (collectives and per-iteration exchanges), not the payload
volume - reducing rounds is the next lever.

Outputs are bit-identical by construction (h5diff clean in serial and at
4 and 8 ranks); tests 350/350 serial, 24/24 MPI (2/3/4 ranks).
@gouarin
gouarin force-pushed the mpi-comm-reduction branch from 14877b2 to c8be3ff Compare July 22, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant