Optional delayed (rank-k) Green's function updates - #646
Conversation
Batches the rank-d corrections of accepted single spin flips into panels X, Y and flushes them into the Green's function with one ZGEMM every k columns, replacing k memory-bound level-2 updates with one level-3 call. The block, rows and columns the vertex loop actually reads are reconstructed on demand, so no accessor needs the full matrix. The factored region spans one time slice's sequential loop only, opening after the hopping propagator and closing before the slice is left, so stabilization, measurement and the global updates are unchanged. The vertex conjugation passes through the factorisation exactly and is applied to the panels by Op_Wrap_panels. Disabled by default. ALF_DELAY_K selects the depth: a positive integer fixes it, "auto" measures it at start-up, "formula" takes the closed form. The resolved depth and how it was chosen are written to info.
… path 36 drives the three panel accessors against an explicitly formed G_stale + X*Y^T, including the edges a whole-run comparison cannot reach: an empty panel, one append short of the flush, and the partial panel left at close. 37 is the drift guard on Op_Wrap_panels, which mirrors sixteen branches of Op_Wrapup/Op_Wrapdo and would diverge silently if either were edited. It sweeps field type, rank, N_type, side, diagonality and the time-dependent coupling, so the ZSLGEMM arms a diagonal-only Hamiltonian never reaches are executed here. 38 composes the two: reads, appends, flushes and conjugations interleaved as Wrapgr_mod interleaves them, at ranks 1 to 4 and both diagonalities, against an immediate arm applying the same updates.
Adds a subsection to the updating chapter covering the factored form, why the conjugation passes through it, the scope of the factored region and the four settings of ALF_DELAY_K. The changelog entry records the two lines the feature adds to every run's info file, which is the only thing an existing setup can notice.
Corrects a factual error (the auto probe takes the largest candidate within 5% of the cheapest, not simply the cheapest) plus several wording/typo fixes in the delayed (rank-k) update writeup.
The panels serve the sequential vertex loop and nothing else, so holding them for a run that never enters that loop is waste: Langevin sets Sequential to false, and the two panels are Ndim*(k+dmax) per flavour that is then never read. Wrapgr_delay_alloc therefore moves out of the block beside Wrapgr_alloc and behind get_sequential(), to just after check_update_schemes_compatibility -- the first point at which Sequential is final, since Langevin turns it off above. A Langevin run now allocates nothing and its info file records the delay as off, even with ALF_DELAY_K set.
delay_probe holds a scratch that grows as Ndim**2, and it is a timing, so running it per rank is wrong twice over. A fully occupied node carries one such scratch per rank, and the stat= fallback to the formula does not save it, since overcommit hands out the memory and the OOM killer collects later. Ranks measuring at once also contend for the very memory system the probe is trying to measure. Rank 0 of Group_Comm now resolves the depth and broadcasts it, through a new delay_set_depth that imposes both the depth and the source string. delayed_update_mod stays free of MPI; Group_Comm rather than MPI_COMM_WORLD because Sequential is broadcast over the group, so the caller's gate on it is uniform across the collective, and because one probe per simulation is the right scope under tempering. The group then runs at one depth, which is also what makes the single line in the info file mean anything.
delay_dealloc reset kmax and delay_active but left panel_w, ndim_s, nfl_s, k_resolved, depth_imposed and delay_source holding the shape of the allocation just released. Nothing reads them today -- alloc and dealloc are called once each per run -- so this buys symmetry rather than a fix: a second allocation cannot inherit the first one's shape. The parsed ALF_DELAY_K is deliberately left cached. It describes the request, not the allocation.
Upgrade2 carried three pairs of near-identical branches, the delayed arm differing from the immediate one only in where it read the Green's function. Each pair is now one path: the d x d block, the d rows behind v and the d columns behind the rank-d update are read into Gblk, g_rows and g_cols once -- off the panels under the delay, off GR otherwise -- and everything downstream reads those and not GR. That drops the duplicated Zarr construction, and with it a second copy of its shape bug: Zarr is d x d but was assigned x_v(P(1:d), :), which is d x Op_dim and so nonconforming wherever the flavours differ in N_non_zero. Now assigned x_v(P(1:d), 1:d). The immediate path is left unchanged bit for bit, including its branch on size(P,1) rather than N_non_zero, which is kept so that the arithmetic order of the rank-1 shortcut does not move.
Tests 36 to 38 drive delayed_update_mod's own routines. None of them reaches Upgrade2's delayed branch, where those routines meet the Woodbury chain -- the reconstructed rows feed v and the reconstructed columns feed the rank-d update, and a mistake there yields a plausible Markov chain rather than an obviously wrong one. Only running the sampler catches that. 39-delayed-vs-immediate runs it twice from one set of seeds, delay off and then ALF_DELAY_K=8, and requires confout_0 to be identical byte for byte. Up to Metropolis near-ties the two schemes accept the same flips, so this is an equality and not a tolerance, and confout is the whole state of the chain. The test reads the depth back out of the info file and fails if it is not 8, so it cannot pass by silently having run the immediate path twice. Two parameter sets, both deliberately tiny: Hubbard for the rank-1 vertices a production run takes on every accepted flip, tV for the rank-2 ones that exercise the Woodbury chain, Zarr and the ZGEMM arm. Opt-in behind ALF_TEST_DELAYED, since unlike the Prog.tests it needs Prog/ALF.out and a plain run directory -- a TEMPERING binary expects Temp_* and would fail here for reasons of its own -- so the existing matrix is untouched and a separate short CI job sets the variable.
Reword the derivation and the ZSLGEMM conversion note, and state the requirement to keep the sixteen mirrored branches in step with Op_Wrapup/Op_Wrapdo as such, pointing at 37-delayed-wrap as the check rather than asserting the mirroring has been verified. Also corrects the stated cost. ZSLGEMM applies an N x N block to the N rows named by Op%P across every column, so the wrap of the panels is O(Op%N**2*ncols), not O(Op%N*ncols).
|
@jonasschwab Can we trigger the pipeline on the pull request? As far as I can see, it didn't run automatically and I don't see a way to trigger it manually. |
I guess the pipeline didn't run because the pull request merges from another repo. So we would have to look at the runs in Sasha's repo: https://github.com/SashaGavrishev/ALF/actions Alternatively, Sasha could push the to-be-merged branch directly to this repo (or I could do it myself, if you want, just a matter of adding a remote in one's local git repo) if we add him as a member. I recently changed the repo config such that all members can push to unprivileged branches. |
I don't think Sasha, or other external contributors, have push rights. We were looking into this a couple of weeks/months ago. Sure, either of us can pull from Sasha, and then create a new branch here. For the long run, we should try to streamline this somehow. |
Are you sure it's not because of the CI configuration? I see here: ALF/.github/workflows/tests.yml Lines 1 to 9 in f81b5cb which has no pull request trigger? (i.e. no |
Ensures PRs get CI feedback before merge, not just after push to a branch.
I guess that might also work. Although it's a bit unfortunate that the There's also test-branch.yaml, test-vs-ed.yaml and tests-broad.yml which are currently only set to |
Mirrors the GitLab test_delayed_update job so the new sampler-level comparison test (testsuite/delayed.tests) runs on GitHub CI too.
Okay that makes sense, I can add
Yeah I see the dilemma for sure; I suppose you're thinking it would be too much to have these on every single pull request? |
| # ====================== | ||
| # Delayed update | ||
| # ====================== | ||
|
|
||
| # The delayed update has to walk the same Markov chain as the immediate one. | ||
| # The compiled tests under Prog.tests cover delayed_update_mod's own routines; | ||
| # only running the sampler covers Upgrade2's delayed branch, where those | ||
| # routines meet the Woodbury chain. One build, four short runs, a byte | ||
| # comparison -- kept out of the matrix above because it needs a plain build and | ||
| # a plain run directory, and because the property is compiler independent. | ||
| test_delayed_update: | ||
| name: Delayed vs immediate update | ||
| runs-on: ubuntu-latest | ||
| container: | ||
| image: ghcr.io/alf-qmc/alf-container/alf-requirements/bookworm | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| - name: Build & compare delayed vs immediate updates | ||
| shell: bash | ||
| run: | | ||
| . ./configure.sh GNU noMPI | ||
| make all | ||
| mkdir "$ALF_DIR/testsuite/tests" | ||
| cd "$ALF_DIR/testsuite/tests" | ||
| ALF_TEST_DELAYED=1 cmake .. | ||
| ALF_TEST_DELAYED=1 ctest -VV -R delayed-vs-immediate | ||
|
|
There was a problem hiding this comment.
I'd add this test to action.yml instead. That's much cleaner, portable and scalable.
Run it from the shared action's noMPI leg instead of a dedicated job, mirroring how the other unit tests are already wired through alf_test.
An HDF5 build writes confout_0.h5 instead of confout_0. HDF5 tracks per-object timestamps by default, so two files holding identical field data still differ byte-for-byte -- cmp -s was failing every HDF5 leg even when the physics matched. h5diff compares dataset contents and ignores that metadata.
The delayed-vs-immediate test needs h5diff to compare confout_0.h5 on HDF5 legs, but CI containers don't ship HDF5 command-line tools -- h5diff: not found. ALF already vendors its own HDF5 build via configure.sh rather than relying on one being preinstalled; pick up h5diff from $HDF5_DIR/bin, which configure.sh leaves set after sourcing.
|
Okay so it looks like the only test that is failing is due to a failed Docker pull |
Resolves four conflicts: - observables_mod: take upstream's 9a0371a, which supersedes our Obs_row fix for the same gfortran/arm64 descriptor fault. - Operator_mod: add/add at the end of the module; keep both our Op_Wrap_panels and upstream's Op_equal / Op_V_is_symmetric. - configure.sh: keep the PKS targets, add upstream's HELMA, and take upstream's GNUDEVFLAGS including -pedantic, re-adding -O0 -ggdb. - Prog.tests: upstream landed 28-35, so the delayed tests renumber to 36/37/38, matching the ALF-QMC#646 branch.
Takes delayed_update_mod, upgrade_mod, Wrapgr_mod and control_mod from the pull-request branch, which carries fixes this tree never had: delay_set_depth resolves the auto depth on one rank and broadcasts it, the panels are allocated only when sequential updates run, delay_dealloc restores pre-alloc state, and the Green function views are read once. main.F90 moves Wrapgr_delay_alloc to match, after Sequential is final. The instrumentation comes back behind one switch. instrument_mod holds the parser the update timer, the hop timer and the near-tie counter each used to repeat; ALF_INSTRUMENT gates all three and defaults off, so a stock build adds nothing to info. The near-tie hoist rewrites the accept/reject line and no runtime test can undo that, so it also sits behind an ALF_INSTRUMENT macro that ALF_NO_INSTRUMENT=1 removes. Toggling the macro needs make clean: make keys on source timestamps, not on flags.
This pull request seeks to add an optional delayed update scheme for only the sequential single-spin-flip sampler as a way to increase performance, especially when the memory system is under load, as it is when, for instance, one uses an HPC node that is fully occupied. The scheme follows F. Sun and X. Y. Xu, Phys. Rev. B 109, 235140 (2024).
Within one time slice we can hold the Green's function in a factorised form
so that on an accepted spin flip proposal, rather than updating the whole matrix we append the rank-$d$ factors of that update as a column pair to the "panels" $X$ and $Y$ . The panels are flushed into $G$ with a single level-3 $k$ accepted columns. This trades the level-2 work of the immediate update, which is memory bandwidth bound, for level-3 work.
ZGEMMeveryThe feature as implemented is off by default and is controlled through an environment variable
ALF_DELAY_K. WhilstALF_DELAY_Kcan be set manually, for instance via personalised benchmarks, we also ship an "auto" mode which via a short numerical probe seeks to select an optimal delay depth for the current system size and computational environment.Implementation
The scheme is introduced via a new module:$N_{\dim} \times (k + d_{\max})$ per flavour, where $d_{\max}$ is the widest wrap support in the model,
Prog/delayed_update_mod.F90. It holds two panels of shapemaxval(Op_V%N).The public surface is small:
delay_openanddelay_closebracket a factorised region.delay_block,delay_rowanddelay_colreturn thedelay_appendapplies a rank-when the panel is full.
delay_wrapconjugates the panels with a vertex operator.The sixteen logical branches of
Op_WrapupandOp_Wrapdoeach need a mirror version for when we are using delayed updates.Op_Wrap_panelsis therefore added toOperator_mod, which applies the vertex updates to the Green's function in its factorised form:A factorised region brackets the sequential vertex loop in
WRAPGRUPandWRAPGRDOand nothing else, so stabilisation, measurement and global moves always see an ordinary Green's function.ALF_DELAY_Kchooses amongst the following cases:0: delayed updates are off, the immediate path runsformula:nint(sqrt(2*Ndim)), clamped toNdimbelow 8 getsNdimauto: a runtime probe times the twoNdimand takes the argmin, falling back to the formula if the measurement failsThe panels$X$ and $Y$ are allocated only when sequential updating is actually enabled.
Behaviour under MPI
Under MPI the probe runs on rank 0 of
Group_Command the result is broadcast. This is to avoid oversaturating the memory by giving each worker its own probe and thus effectively ruining the benchmark / running out of memory.The module itself contains no MPI and needs no bespoke MPI functionality.
Other code changes
upgrade_mod:Upgrade2reads the block, rows and columns of the Green's function into three views once, off the panels under the delay and offGRotherwise, and one code path follows for both schemes. This also fixes a pre-existing shape bug in theZarrslice, which wasOp_dimagainst aN_non_zero.Wrapgr_mod: allocation and deallocation of the panels, the open and close calls, the paireddelay_wrapcalls, and two assertions that no factorised region is open inWrapgr_PlaceGRorWrapgr_Random_update.main.F90: gated allocation and a rank-guarded log of the resolved depth.control_mod: the two new info file lines which detail the delay depth and how it was picked.Documentation
Documentation/updating.tex: a new subsubsection is added describing the scheme.CHANGELOG.md: entry.Testing
Four new tests are added
36-delayed-panelsdrives the panel accessors against an explicit matrix.37-delayed-wrapdrivesOp_Wrap_panelsover all sixteen wrap branches.38-delayed-updatecomposes the two, interleaving reads, rank-N_Typevalues. The round count is chosen so that every rank both crosses the flush threshold and ends up holding a partial panel, so the trailing flush is exercised too.39-delayed-vs-immediateruns the sampler itself twice from one set of seeds delay off and thenALF_DELAY_K=8, and requiresconfout_0to be identical byte for byte. This is the only test that reachesUpgrade2's delayed branch. Two parameter sets are used, one with rank-1 vertices and one with rank-2.The first three tests run in the existing suite. The fourth is added as an opt-in behind
ALF_TEST_DELAYED; it needsProg/ALF.outand a plain run directory. A short CI job,test_delayed_updatein.gitlab-ci.yml, sets that variable and runs it, leaving the existing test matrix untouched.Beyond the suite, the two schemes were compared end to end on twelve configurations: Hubbard including the projector, continuous fields, no-Mz and no-Symm/no-Checkerboard variants, plus Kondo, tV, LRC, Z2_Matter, Spin_Peierls, Hubbard_Plain_Vanilla, and LRC with global-in-tau moves.
confout_0was identical in every case, at fixed depths and underautoandformula, and acceptance rates matched to all printed digits. The immediate path was separately confirmed unchanged bit for bit against the binary built before these changes.Performance (rough laptop benchmarks)
Hubbard on an$L \times L$ square lattice,
Mz,Beta=2,Dtau=0.1,Nwrap=10,Ltau=0, checkerboard and symmetrisation on,ALF_DELAY_K=auto. Apple M4 Pro, 10 performance cores, 48 GB, macOS 26.6.2, GNU Fortran 16.1.0 at-O3 -ffast-math,OMP_NUM_THREADS=1. Solo is one process; packed is ten concurrent processes, one per performance core, each in its own run directory.Figures are the speedup over the immediate update in per-sweep cost, obtained by differencing two sweep counts so that the fixed setup cost drops out. The values are the mean of two repeats. The last column is the depth the probe settled on in the first repeat; it chose differently in the second in three of the eight cells, without really changing the timings.
NdimAt every size, a saturated compute environment benefits the most from delayed updates. However, at$N_{\dim} = 64$ performance in delayed updates was worse, likely because the matrix was already small enough to fit in cache and the overhead of factorising the Green's function becomes too expensive.
Safety
Nothing in this contribution affects the state of the random generator; in principle delays "on" vs "off" should reproduce the same Markov chain up to a small caveat – Metropolis near-ties. Delayed updates reassociate the arithmetic at the stage of the Green's function updates and hence will introduce a relative numerical error. This has the potential to cause branching at the level of a Metropolis accept / reject steps and hence produce a new chain. However, across the twelve differential configurations described above this has yet to be observed. The factorised form is in any case confined to a single time slice, so this error cannot accumulate across slices and stabilisation, measurement and global moves are untouched.
Code provenance
The code in this pull request / branch was first scaffolded by an LLM to a brief and then reviewed by a human.