ci(rccl): make the perf-regression gate refuse to answer when it did not measure

ci/rccl_perf_gate/README.md

@@ -0,0 +1,27 @@
+# rccl_perf_gate
+
+Slurm submission/polling/reporting glue for the RCCL paired A/B performance
+regression gate used by `ROCm/rocm-systems`'s
+[`rccl_perf_regression.yml`](https://github.com/ROCm/rocm-systems/blob/main/.github/workflows/rccl_perf_regression.yml)
+GitHub Actions workflow.
+
+The workflow's self-hosted runner invokes these scripts directly:
+
+- `sbatch/rccl_build.sbatch`, `sbatch/run_rccl_build.sh` — build RCCL (via
+  `cvs-sbatch`) as a Slurm job.
+- `submit_and_poll.sh`, `sbatch/rccl_ab.sbatch`, `sbatch/run_rccl_ab.sh` —
+  submit the paired A/B regression job (`cvs/tests/rccl/rccl_ab_regression.py`),
+  poll it to completion, and map its exit code to a CI-gatable result.
+- `format_report.py` — render the A/B run's JSON result into a Markdown
+  summary for the workflow's job summary / PR comment.
+
+All scripts honor an `RCCL_CI_ROOT` env override (default `/it-share/rccl-ci`)
+so they aren't tied to one cluster's NFS layout.
+
+## Status
+
+This is a stopgap. It exists because CVS does not yet submit and manage Slurm
+(or Kubernetes) jobs natively — these scripts are thin bash wrappers around
+`sbatch`/`squeue` bridging that gap. Once CVS gains native scheduler
+integration, this directory should be retired in favor of driving the A/B
+regression test directly through CVS.

ci/rccl_perf_gate/configs/README.md

@@ -0,0 +1,51 @@
+# Perf-gate configs
+
+These are the configs that decide what the gate measures and what counts as a
+regression. They used to exist **only** at `/it-share/rccl-ci/configs/` on NFS,
+which meant the gate's decision boundary had no history: a threshold could be
+widened by hand at 2am and nothing would record that it happened, who did it, or
+what it was before. A green check is only as trustworthy as the numbers behind
+it, so those numbers are now version-controlled.
+
+## Which file does what
+
+| file | used by | role |
+|---|---|---|
+| `ci_detect_prod.json` | the PR gate (`rccl_perf_regression.yml`, default `config` input) | reference-vs-candidate detection: test matrix, repeats, thresholds, timeouts |
+| `ci_control.json` | calibration / control (A=A) runs | derives the noise floor that the detect thresholds are set against |
+
+## NFS is still the live copy
+
+Nothing reads from this directory at runtime. `/it-share/rccl-ci/configs/` remains
+the deployment target, because the workflow, the sbatch scripts and hand-run
+`workflow_dispatch` invocations all pass absolute paths into it.
+
+So this directory is a **source of truth that must be kept in sync by hand**:
+
+```bash
+# after editing a config here
+scp cvs/ci/rccl_perf_gate/configs/ci_detect_prod.json \
+    tensorwave-slurm-rccl:/it-share/rccl-ci/configs/ci_detect_prod.json
+
+# to check for drift
+ssh tensorwave-slurm-rccl 'md5sum /it-share/rccl-ci/configs/ci_detect_prod.json'
+md5sum cvs/ci/rccl_perf_gate/configs/ci_detect_prod.json
+```
+
+Wiring the readers to pull straight from the repo checkout would remove the
+manual step, but it changes where a live gate loads its config from, so it is
+deliberately left as a follow-up rather than folded into a robustness pass.
+
+## Editing thresholds
+
+Don't hand-tune `thresholds`. Run a control (A=A) calibration, which writes
+`configs/ab_derived_thresholds.json` via `median + k*MAD`, and let the detector
+pick it up. `max_thresholds` is the ceiling that stops a noisy calibration from
+loosening the gate into uselessness — that one is a policy decision and *is*
+meant to be edited by hand.
+
+`_comment` fields inside the configs record why individual collectives are
+skipped. Read them before re-enabling anything: `alltoall_perf` is excluded
+because pooling it inflated the derived large-tier threshold by 60-100x, which
+would have blinded the gate for every other collective. Per-collective
+thresholds now make re-enabling it possible, but only after a fresh calibration.

ci/rccl_perf_gate/configs/ci_control.json

@@ -0,0 +1,102 @@
+{
+  "_comment": "Control (A=A) calibration counterpart to ci_detect_prod.json. Everything that affects measurement -- size sweep, collectives, skip_keys, repeats, timeouts, min_bandwidth_floor -- MUST match ci_detect_prod.json, otherwise the derived thresholds describe a different experiment than the one being gated. min_bandwidth_floor in particular: a scalar 0.5 GB/s floor here would leave the small tier calibrated on nothing while detect measures it. Thresholds derived by this run are published to configs/ab_derived_thresholds.json only if the run clears its own trustworthiness checks. | RCCL CI calibration / A=A control. Derives ab_derived_thresholds.json and MUST report 0 regressions. Both ref+cand point at builds/_fixed/lib (rebuilt 2026-07-14 from rocm-systems develop HEAD 8a7d08d925, which fixes the gfx950 DDA alltoall segfault, the large-message (>=1G) alltoall OOB/abort/hang, and the alltoall bfloat16 correctness bug -- see CI_PIPELINE_PLAN.md caveats #1/#1b; confirmed via node_health_test.sh smallmsg+largemsg+bf16 sweeps, 0 wrong across all). RCCL_DDA_ENABLE override REMOVED from ainic_rccl_env.sh (RCCL default DdaEnable=1 applies). alltoall_perf re-enabled (removed from skip_keys). Recalibrate on any HW/ROCm/RCCL-build/matrix/node/env change. | UPDATE 2026-07-13: alltoall correctness bugs (DDA segfault c59673fc, bf16 correctness, large-msg >=1G OOB/abort/hang) confirmed FIXED on develop HEAD 8a7d08d925 -- lib rebuilt/repinned, RCCL_DDA_ENABLE override removed from ainic_rccl_env.sh entirely. 2026-07-13 (superseded, kept for the audit trail): control-mode calibration job 12389 measured ~5-6x A=A swings for alltoall_perf in the 128MB-4G band, unique among the 5 gated collectives, at a boundary aligning with the 64MB DDA_THRESHOLD. With a single shared large-tier threshold that inflated the derived value ~60-100x (0.126 -> 8.41), which would have blinded the gate for ALL 5 collectives, so alltoall was skip_keys'd out of comparison pending root-cause or per-collective thresholds. Note the claim in that entry that it 'stays in rccl_collective (runs, for visibility)' was never true: skip_keys is a pytest.skip, so the sweep did not execute at all and no data was archived. || UPDATE 2026-08-12: alltoall_perf RE-ENABLED (skip_keys emptied). Both halves of the 2026-07-13 rationale have changed. (a) The unblocking condition it named is met: thresholds_by_collective is derived per collective and resolved per collective at detect time, and max_thresholds bounds any one collective's contribution, so a noisy collective can no longer widen the gate for the others. The only pooled value alltoall still moves is __default__ (large 0.0300 -> 0.0498), which no gated collective reads while all 5 have their own entry. (b) The instability itself does not reproduce: job 16368 (2026-08-12, 4 nodes, 7 repeats, 10 groups, 460 keys) measured the 128MB-4G band at max/min 1.08-1.14 per size with cv_median 0.022, against the ~5-6x reported in July, and returned 0 A=A false positives -- also 0 when the three gates were replayed offline against the per-collective table production actually uses. alltoall is no longer the noisiest gated collective: broadcast_perf-d=float has more >2x keys (3) than alltoall (1) and a higher small-tier cv_median (0.085 vs 0.031). What changed between July and August is not isolated here; the SDK dist was normalised (DMA-BUF symlink chain restored) in that window, which is a plausible but unproven cause. CAVEAT: AllToAll mid and large derive above the max_thresholds ceiling (0.0927 vs 0.08, 0.0748 vs 0.06), so calibration cannot widen them further. If its noise grows the gate will emit false positives rather than silently pass regressions -- the right failure direction, but re-exclude if that starts happening. Worst A=A rel_drop measured was 0.0161 mid / 0.0165 large, so there is real margin today.",
+  "rccl": {
+    "mpi_params": {
+      "no_of_nodes": "4",
+      "no_of_local_ranks": "8",
+      "mpi_pml": "ob1",
+      "mpi_dir": "/it-share/ompi-5.0.8",
+      "mpi_oob_port": "eno0",
+      "ucx_tls": "tcp"
+    },
+    "env_source_script": "/it-share/rccl-ci/cvs-sbatch/env/ainic_rccl_env.sh",
+    "rccl_test_params": {
+      "rccl_tests_dir": "/it-share/rccl-ci/rccl-tests-2.30.4/bin",
+      "start_msg_size": "1024",
+      "end_msg_size": "4G",
+      "step_function": "2",
+      "threads_per_gpu": "1",
+      "warmup_iterations": "10",
+      "no_of_iterations": "20",
+      "no_of_cycles": "1",
+      "check_iteration_count": "1",
+      "rccl_timeout": "360",
+      "per_collective_timeout_sec": 360,
+      "output_algo_proto_channels": false
+    },
+    "cvs_params": {
+      "cluster_snapshot_debug": "False",
+      "nic_model": "ainic",
+      "verify_bus_bw": "False",
+      "verify_bw_dip": "False",
+      "verify_lat_dip": "False",
+      "cvs_exec_timeout": "7200",
+      "rccl_result_file": "/tmp/rccl_ci_control.json"
+    },
+    "gpu_cleanup": {
+      "enabled": true,
+      "kill_gpu_pids": true,
+      "kill_containers": false,
+      "use_sudo": false
+    },
+    "retry": {
+      "max_retries": 2,
+      "backoff_sec": 15
+    },
+    "rccl_collective": [
+      "all_reduce_perf",
+      "reduce_scatter_perf",
+      "all_gather_perf",
+      "broadcast_perf",
+      "alltoall_perf"
+    ],
+    "data_types": [
+      "float",
+      "bfloat16"
+    ],
+    "ab_regression": {
+      "repeats": 7,
+      "control_mode": true,
+      "skip_keys": [],
+      "safety_factor": 2.0,
+      "adjacency_min_run": 2,
+      "min_repeats": 2,
+      "min_bandwidth_floor": {
+        "small": 0.005,
+        "mid": 0.05,
+        "large": 0.5
+      },
+      "metric": "busBw",
+      "higher_is_better": true,
+      "thresholds": {
+        "small": 0.15,
+        "mid": 0.08,
+        "large": 0.06
+      },
+      "tier_boundaries": {
+        "small_max_bytes": 1048576,
+        "mid_max_bytes": 67108864
+      },
+      "output_dir": "/it-share/rccl-ci/ab_artifacts",
+      "reference": {
+        "label": "ref",
+        "rccl_tests_dir": "/it-share/rccl-ci/rccl-tests-2.30.4/bin",
+        "ld_library_path": "/it-share/rccl-ci/builds/_fixed/lib:/it-share/ompi-5.0.8/lib:/it-share/rccl-ci/rocm_devel/lib:/it-share/rccl-ci/lib/libionic"
+      },
+      "candidate": {
+        "label": "cand",
+        "rccl_tests_dir": "/it-share/rccl-ci/rccl-tests-2.30.4/bin",
+        "ld_library_path": "/it-share/rccl-ci/builds/_fixed/lib:/it-share/ompi-5.0.8/lib:/it-share/rccl-ci/rocm_devel/lib:/it-share/rccl-ci/lib/libionic"
+      },
+      "max_thresholds": {
+        "small": 0.15,
+        "mid": 0.08,
+        "large": 0.06
+      },
+      "mad_k": 3.0,
+      "circuit_breaker_failures": 2,
+      "require_balanced_samples": true,
+      "max_inconclusive_frac": 0.1
+    }
+  }
+}

ci/rccl_perf_gate/configs/ci_detect_prod.json

@@ -0,0 +1,102 @@
+{
+  "_comment": "RCCL per-PR perf-regression detection. Ref+cand both use _fixed/lib (rebuilt 2026-07-14 from rocm-systems develop HEAD 8a7d08d925 -- fixes gfx950 DDA alltoall segfault, large-message (>=1G) alltoall OOB/abort/hang, and alltoall bfloat16 correctness bug). BUILD_RCCL=1 overwrites reference/candidate in-allocation for PRs touching projects/rccl. alltoall_perf re-enabled (removed from skip_keys); RCCL_DDA_ENABLE override removed from ainic_rccl_env.sh. | UPDATE 2026-07-13: alltoall correctness bugs (DDA segfault c59673fc, bf16 correctness, large-msg >=1G OOB/abort/hang) confirmed FIXED on develop HEAD 8a7d08d925 -- lib rebuilt/repinned, RCCL_DDA_ENABLE override removed from ainic_rccl_env.sh entirely. 2026-07-13 (superseded, kept for the audit trail): control-mode calibration job 12389 measured ~5-6x A=A swings for alltoall_perf in the 128MB-4G band, unique among the 5 gated collectives, at a boundary aligning with the 64MB DDA_THRESHOLD. With a single shared large-tier threshold that inflated the derived value ~60-100x (0.126 -> 8.41), which would have blinded the gate for ALL 5 collectives, so alltoall was skip_keys'd out of comparison pending root-cause or per-collective thresholds. Note the claim in that entry that it 'stays in rccl_collective (runs, for visibility)' was never true: skip_keys is a pytest.skip, so the sweep did not execute at all and no data was archived. || UPDATE 2026-08-12: alltoall_perf RE-ENABLED (skip_keys emptied). Both halves of the 2026-07-13 rationale have changed. (a) The unblocking condition it named is met: thresholds_by_collective is derived per collective and resolved per collective at detect time, and max_thresholds bounds any one collective's contribution, so a noisy collective can no longer widen the gate for the others. The only pooled value alltoall still moves is __default__ (large 0.0300 -> 0.0498), which no gated collective reads while all 5 have their own entry. (b) The instability itself does not reproduce: job 16368 (2026-08-12, 4 nodes, 7 repeats, 10 groups, 460 keys) measured the 128MB-4G band at max/min 1.08-1.14 per size with cv_median 0.022, against the ~5-6x reported in July, and returned 0 A=A false positives -- also 0 when the three gates were replayed offline against the per-collective table production actually uses. alltoall is no longer the noisiest gated collective: broadcast_perf-d=float has more >2x keys (3) than alltoall (1) and a higher small-tier cv_median (0.085 vs 0.031). What changed between July and August is not isolated here; the SDK dist was normalised (DMA-BUF symlink chain restored) in that window, which is a plausible but unproven cause. CAVEAT: AllToAll mid and large derive above the max_thresholds ceiling (0.0927 vs 0.08, 0.0748 vs 0.06), so calibration cannot widen them further. If its noise grows the gate will emit false positives rather than silently pass regressions -- the right failure direction, but re-exclude if that starts happening. Worst A=A rel_drop measured was 0.0161 mid / 0.0165 large, so there is real margin today. | Static thresholds fallback refreshed 2026-07-14 from job 12455 control-mode recalibration (4 collectives, alltoall excluded) -- use_derived_thresholds is True by default so ab_derived_thresholds.json is the live source of truth at detect time; this field is only the fallback if that file is ever missing.",
+  "rccl": {
+    "mpi_params": {
+      "no_of_nodes": "4",
+      "no_of_local_ranks": "8",
+      "mpi_pml": "ob1",
+      "mpi_dir": "/it-share/ompi-5.0.8",
+      "mpi_oob_port": "eno0",
+      "ucx_tls": "tcp"
+    },
+    "env_source_script": "/it-share/rccl-ci/cvs-sbatch/env/ainic_rccl_env.sh",
+    "rccl_test_params": {
+      "rccl_tests_dir": "/it-share/rccl-ci/rccl-tests-2.30.4/bin",
+      "start_msg_size": "1024",
+      "end_msg_size": "4G",
+      "step_function": "2",
+      "threads_per_gpu": "1",
+      "warmup_iterations": "10",
+      "no_of_iterations": "20",
+      "no_of_cycles": "1",
+      "check_iteration_count": "1",
+      "rccl_timeout": "360",
+      "per_collective_timeout_sec": 360,
+      "output_algo_proto_channels": false
+    },
+    "cvs_params": {
+      "cluster_snapshot_debug": "False",
+      "nic_model": "ainic",
+      "verify_bus_bw": "False",
+      "verify_bw_dip": "False",
+      "verify_lat_dip": "False",
+      "cvs_exec_timeout": "7200",
+      "rccl_result_file": "/tmp/rccl_ci_detect.json"
+    },
+    "gpu_cleanup": {
+      "enabled": true,
+      "kill_gpu_pids": true,
+      "kill_containers": false,
+      "use_sudo": false
+    },
+    "retry": {
+      "max_retries": 2,
+      "backoff_sec": 15
+    },
+    "rccl_collective": [
+      "all_reduce_perf",
+      "reduce_scatter_perf",
+      "all_gather_perf",
+      "broadcast_perf",
+      "alltoall_perf"
+    ],
+    "data_types": [
+      "float",
+      "bfloat16"
+    ],
+    "ab_regression": {
+      "repeats": 7,
+      "control_mode": false,
+      "skip_keys": [],
+      "safety_factor": 2.0,
+      "adjacency_min_run": 2,
+      "min_repeats": 2,
+      "min_bandwidth_floor": {
+        "small": 0.005,
+        "mid": 0.05,
+        "large": 0.5
+      },
+      "metric": "busBw",
+      "higher_is_better": true,
+      "thresholds": {
+        "small": 0.15,
+        "mid": 0.08,
+        "large": 0.06
+      },
+      "tier_boundaries": {
+        "small_max_bytes": 1048576,
+        "mid_max_bytes": 67108864
+      },
+      "output_dir": "/it-share/rccl-ci/ab_artifacts",
+      "reference": {
+        "label": "ref",
+        "rccl_tests_dir": "/it-share/rccl-ci/rccl-tests-2.30.4/bin",
+        "ld_library_path": "/it-share/rccl-ci/builds/_fixed/lib:/it-share/ompi-5.0.8/lib:/it-share/rccl-ci/rocm_devel/lib:/it-share/rccl-ci/lib/libionic"
+      },
+      "candidate": {
+        "label": "cand",
+        "rccl_tests_dir": "/it-share/rccl-ci/rccl-tests-2.30.4/bin",
+        "ld_library_path": "/it-share/rccl-ci/builds/_fixed/lib:/it-share/ompi-5.0.8/lib:/it-share/rccl-ci/rocm_devel/lib:/it-share/rccl-ci/lib/libionic"
+      },
+      "circuit_breaker_failures": 2,
+      "max_thresholds": {
+        "small": 0.15,
+        "mid": 0.08,
+        "large": 0.06
+      },
+      "max_inconclusive_frac": 0.1,
+      "require_balanced_samples": true,
+      "mad_k": 3.0
+    }
+  }
+}