Add new application: petsc-gmres

[william-dawson]

A PETSc KSP (GMRES + GAMG) benchmark, verified on Rikyu, Fugaku, and R-CCS Cloud's DGX Spark. [code:petsc-gmres]

[yoshifuminakamura]

Thanks for the detailed validation notes. One current blocker is DCO: the check is failing on this draft PR. Could you please add Signed-off-by lines to the commits that are missing them, or otherwise update the branch so the DCO check passes?

[william-dawson]

GPU scaling re-measured under the iterate-only FOM

Per Suzuki-san's recommendation, the FOM now measures only the GMRES iterate (ksp_iter_time_s), with KSPSetUp called explicitly before the timed KSPSolve so GAMG setup is excluded. Re-measured on Rikyu (GB200 NVL4, 2026-08-17):

GPUs	solve_time_s (old, setup-inclusive)	ksp_iter_time_s (new, iterate-only)	relative L2
1	1.55	0.129	0.0443346
2	5.81	0.211	0.0342875
4	6.04	0.183	0.03154

Setup was 91-97% of the old FOM — even more dominant than the 70% -log_view reported. Under the iterate-only FOM, 1 GPU still wins (0.129s), but the gap narrowed from ~4x to ~1.5x. Notably, 4 GPUs now beats 2 GPUs (0.183 vs 0.211), reversing the old near-tie — the iterate does parallelize, just not enough at 943k rows (~236k rows/GPU at 4 GPUs) to overtake 1 GPU. A larger matrix is the honest next test of whether multi-GPU can win under the iterate-only FOM.

Correctness held throughout (L2 in the expected 0.03-0.04 band, bit-identical to the old runs at each GPU count).

[william-dawson]

stokes2 matrix — full three-machine results (iterate-only FOM)

Switched the benchmark matrix from audikw_1 (943k rows, SPD structural mechanics) to stokes2 (4.26M rows, Stokes flow saddle-point system, generated via Gmsh/FreeFEM by Suzuki-san). Already in PETSc binary format (FreeFEM ObjectView = MatView), so no mtx2petsc conversion needed. KSPSolve is ~10× KSPSetUp (confirmed via -log_view), so the iterate dominates — the iterate-only FOM is the right metric.

Rikyu (GB200 NVL4, GPU, 1 rank/GPU)

GPUs	ksp_iter_time_s	iterations	relative L2	speedup
1	1.497	232	0.2516	1.00x
2	0.932	200	0.2566	1.61x
4	0.683	193	0.2278	2.19x

Multi-GPU now wins — at ~1M rows/GPU there is enough work to amortize communication. Compare with audikw_1 under the same FOM: 1 GPU 0.129s, 4 GPU 0.183s (1.41× slower). The larger matrix flips the conclusion.

Fugaku (A64FX, multi-node CPU, 48 ranks/node, f-pt queue)

nodes	ranks	ksp_iter_time_s	relative L2	speedup
1	48	44.881	0.2663	1.00x
2	96	28.532	0.2582	1.57x
4	192	16.777	0.2317	2.67x
8	384	15.959	0.2641	2.81x

Clean scaling through 4 nodes (2.67× at 4× ranks), then nearly flat 4→8 (2.81× at 8× ranks) — communication wall hits ~192 ranks for this problem size on A64FX.

DGX Spark (GB10 Blackwell, 1 GPU)

machine	GPU	ksp_iter_time_s	relative L2
Rikyu	B200	1.497	0.2516
DGX Spark	GB10	21.430	0.2516

Correctness bit-identical (L2 0.251638 on both). 14.3× slowdown on GB10 vs B200 — larger than audikw_1's 5.2×, since the larger matrix stresses memory bandwidth more and GB10 has a smaller bandwidth budget.

Summary

All three machines validated via test_submit.sh with the stokes2.dat data staged at the per-system paths in run.sh. The f-pt queue was used for the Fugaku multi-node runs (wider node range than small); the committed list.csv/system.csv stay on small.

[yoshifuminakamura]

@william-dawson #134 has been merged into develop/main and addresses the common scheduler/CI side that came up while reviewing this PR.

Could you please rebase #87 onto the latest develop and update it accordingly?

• Fugaku project group selection should now use BK_ALLOCATION_PROJECT_ID, so the BK_PJM_GROUP / scripts/test_submit.sh change in this PR should be dropped.
• The RC_DGXSP MATHLIBS assignment should be fixed; the current glob-in-scalar-assignment form is caught by the new warning-level shellcheck for changed app scripts.
• After rebasing, please rerun CI against the latest develop.

This should keep #87 focused on the petsc-gmres app addition while the common scheduler/CI handling stays in #134.

[yoshifuminakamura]

@william-dawson Thanks for the update. I checked the latest head and the previous blocker around the DGX Spark MATHLIBS glob is fixed. The PR is also based on the current develop, CI is green, and I do not see the large input data included in the PR.

One small cleanup before review/merge: programs/petsc-gmres/src/GMRES-PETSc.cpp still has an old top-of-file comment that refers to audikw_1, MatrixMarket, and .petscbin, while the app has moved to stokes2.dat generated via FreeFEM/ObjectView. Could you update that comment so it matches the current README/run.sh behavior?

If that is the only remaining item on your side, please also mark the PR ready for review / undraft after the cleanup.

[william-dawson]

Updated the top-of-file comment and usage string in GMRES-PETSc.cpp to replace the stale audikw_1 / MatrixMarket / .petscbin references with stokes2 / FreeFEM ObjectView / stokes2.dat, matching the current README and run.sh. Also corrected SPD matrix → Stokes flow saddle-point system and solve wall-time → GMRES-iterate wall-time to reflect the iterate-only FOM. PR is marked ready for review.