diff --git a/ManPage.md b/ManPage.md index 4f834d4d..b69068b4 100644 --- a/ManPage.md +++ b/ManPage.md @@ -607,6 +607,9 @@ Required positionals: `` (one of `llama3-8b`, `llama3-70b`, `llama3-405b` - **`--checkpoint-folder `, `-cf `** Storage location for checkpoint files. Required for `run`. +- **`--hpc`** *(run/configview only)* + Declare an HPC shared-parallel-filesystem environment where the §4.7.1 two-invocation failover callout cannot meet the 30-second budget (the write and read phases are separate scheduler jobs whose node sets cannot be guaranteed identical, and per-invocation MPI + DLIO re-init over a multi-TB tree alone exceeds 30s). Relaxes the §4.7.1 30-second gap check and the 1-or-2 invocation-structure check to warnings, and is recorded in `metadata.json` for auditability. Cold reads must still be ensured by the storage layer / benchmark (remote parallel FS, `--o-direct`, or `posix_fadvise(DONTNEED)` on close). See Rules §4.7.1. + - **`--hosts`, `--exec-type`, `--dlio-bin-path`** Same semantics as the training options of the same name. diff --git a/Rules.md b/Rules.md index 44ca27ec..2830d787 100644 --- a/Rules.md +++ b/Rules.md @@ -520,7 +520,7 @@ root_folder (or any name you prefer) ## 4.7. Storage System Must Be Simultaneously R/W or _Remappable_ -4.7.1. **checkpointCacheFlushValidation** -- Checkpointing models the failure of a client node followed by another client picking up the last checkpoint file written by the failed node for the read phase. In every submission the write phase (10 checkpoint files written) runs first, followed by the read phase (10 checkpoint files read). When the storage system supports the client-to-client handoff transparently — i.e., the read phase can proceed immediately after the write phase without external orchestration — the write and read phases may be executed as a single combined invocation, and no gap check applies. Storage system architectures that require an external callout (e.g., a submitter-provided script) to complete the failover between the writing and reading clients must instead execute the write and read phases as two separate invocations, with the submitter's failover callout occurring between them. A common in-callout activity is clearing a client-side filesystem cache when the total checkpoint size written per client is less than 3× the client node's memory capacity (see ``checkpointing/README.md``); the callout is not limited to that activity. To ensure the callout is a lightweight programmatic step rather than a long-running manual procedure, the validator confirms that the read-phase invocation was launched — i.e., its ``mlpstorage`` process reached the entry point of ``main.py`` — within 30 seconds of the write-phase invocation ending. The gap is measured as ``read.invocation_start_time − write.summary.end_time``, where ``invocation_start_time`` is captured at ``mlpstorage`` process start (before framework startup, MPI spawn, and other unavoidable per-invocation overhead) and ``end_time`` is recorded in the write invocation's ``summary.json``. A negative gap indicates clock skew between the write and read nodes and is reported as such rather than as a causality violation. +4.7.1. **checkpointCacheFlushValidation** -- Checkpointing models the failure of a client node followed by another client picking up the last checkpoint file written by the failed node for the read phase. In every submission the write phase (10 checkpoint files written) runs first, followed by the read phase (10 checkpoint files read). When the storage system supports the client-to-client handoff transparently — i.e., the read phase can proceed immediately after the write phase without external orchestration — the write and read phases may be executed as a single combined invocation, and no gap check applies. Storage system architectures that require an external callout (e.g., a submitter-provided script) to complete the failover between the writing and reading clients must instead execute the write and read phases as two separate invocations, with the submitter's failover callout occurring between them. A common in-callout activity is clearing a client-side filesystem cache when the total checkpoint size written per client is less than 3× the client node's memory capacity (see ``checkpointing/README.md``); the callout is not limited to that activity. To ensure the callout is a lightweight programmatic step rather than a long-running manual procedure, the validator confirms that the read-phase invocation was launched — i.e., its ``mlpstorage`` process reached the entry point of ``main.py`` — within 30 seconds of the write-phase invocation ending. The gap is measured as ``read.invocation_start_time − write.summary.end_time``, where ``invocation_start_time`` is captured at ``mlpstorage`` process start (before framework startup, MPI spawn, and other unavoidable per-invocation overhead) and ``end_time`` is recorded in the write invocation's ``summary.json``. A negative gap indicates clock skew between the write and read nodes and is reported as such rather than as a causality violation. **HPC exemption:** on an HPC shared parallel filesystem, the two-invocation failover callout cannot meet the 30-second budget — the write and read phases are separate batch-scheduler jobs whose node sets cannot be guaranteed identical, and the read invocation's unavoidable per-invocation MPI spawn plus DLIO re-initialization over a multi-TB checkpoint tree alone can take several minutes to reach ``main.py``. A submitter may declare this environment by passing ``--hpc`` to ``mlpstorage ... checkpointing run``. When ``--hpc`` is recorded in the run's ``metadata.json``, the validator relaxes the 30-second gap check and the 1-or-2 invocation-structure check to warnings (so the exemption is visible in the report rather than silently applied). On such systems cold reads are ensured by the storage layer and the benchmark itself — a remote parallel filesystem, ``O_DIRECT`` (``--o-direct``), and/or the benchmark's ``posix_fadvise(POSIX_FADV_DONTNEED)`` on file close — rather than by the callout timing; the submitter remains responsible for ensuring the read phase is served cold from storage. 4.7.2. **checkpointTotalTestDuration** -- The validator must verify that the total test duration starts from the timestamp of the first checkpoint written and ends at the ending timestamp of the last checkpoint read, notably including the "remapping" time. diff --git a/mlpstorage_py/cli/checkpointing_args.py b/mlpstorage_py/cli/checkpointing_args.py index 9fcc7f43..0998d041 100755 --- a/mlpstorage_py/cli/checkpointing_args.py +++ b/mlpstorage_py/cli/checkpointing_args.py @@ -57,13 +57,16 @@ def _add_checkpointing_core_args(parser, command): parser: The subcommand parser to add arguments to. command: The subcommand name ('datasize', 'run', 'configview'). """ - # Set defaults for open-gated attrs so they always exist in the namespace + # Set defaults for open-gated attrs so they always exist in the namespace. + # hpc defaults False here so vars(args) always carries it (recorded in + # metadata.json) even on subcommands that don't expose the flag. parser.set_defaults( loops=1, params='', allow_invalid_params=False, dlio_bin_path=None, checkpoint_folder=None, + hpc=False, ) add_host_arguments(parser) @@ -137,6 +140,32 @@ def _add_checkpointing_core_args(parser, command): ), ) + # --hpc: available for run and configview (not datasize). + # Declares an HPC shared-parallel-filesystem environment where the §4.7.1 + # two-invocation failover callout cannot complete within the 30-second + # budget (the write and read phases are separate scheduler jobs on nodes + # that cannot be guaranteed identical, and per-invocation MPI + DLIO + # re-initialization over a multi-TB checkpoint tree alone exceeds 30s). + # See mlcommons/storage and Rules.md §4.7.1. + if command in ("run", "configview"): + parser.add_argument( + '--hpc', + action='store_true', + default=False, + dest='hpc', + help=( + "Declare an HPC shared-parallel-filesystem environment where " + "the §4.7.1 same-node, <=30s two-invocation failover-callout " + "workflow is infeasible. Relaxes the 30s inter-phase gap and " + "invocation-structure enforcement for this run. Recorded in " + "metadata.json for auditability; on such systems cold reads " + "are ensured by the storage layer / benchmark (e.g. " + "posix_fadvise(DONTNEED) on close, O_DIRECT, or a remote " + "parallel filesystem) rather than by the callout timing, and " + "the submitter remains responsible for cold reads." + ), + ) + # Checkpoint folder required for run only if command == "run": parser.add_argument( diff --git a/mlpstorage_py/cli/help_formatter.py b/mlpstorage_py/cli/help_formatter.py index 6f09fc40..90901883 100644 --- a/mlpstorage_py/cli/help_formatter.py +++ b/mlpstorage_py/cli/help_formatter.py @@ -296,11 +296,17 @@ --hosts/-s HOST... (default: 127.0.0.1) --num-checkpoints-read/-ncr N (default: 10; closed allows 10 or 0) --num-checkpoints-write/-ncw N (default: 10; closed allows 10 or 0) + --hpc (relax §4.7.1 30s gap + invocation + structure for HPC shared filesystems) + MPI_ARGS + CORE_STD Note: closed runs use 10/10 by default. Use 10/0 then 0/10 in two invocations when a cache flush is required between phases (see Rules.md §4.7.1 and checkpointing/README.md). + Note: --hpc declares an HPC shared-parallel-filesystem environment where + the two-invocation failover callout cannot meet the 30s budget; it + relaxes the §4.7.1 gap + invocation-structure checks to warnings and + is recorded in metadata.json (see Rules.md §4.7.1). Closed rank constraints by model: llama3-1t: 8 or 1024 diff --git a/mlpstorage_py/submission_checker/checks/checkpointing_checks.py b/mlpstorage_py/submission_checker/checks/checkpointing_checks.py index a855c48b..8552a8b3 100644 --- a/mlpstorage_py/submission_checker/checks/checkpointing_checks.py +++ b/mlpstorage_py/submission_checker/checks/checkpointing_checks.py @@ -114,7 +114,29 @@ def _iter_valid_files(self): ) continue yield summary, metadata, timestamp - + + def _hpc_exempt(self): + """True if any run in this workload declared the §4.7.1 HPC exemption. + + The exemption is opted into at run time with ``mlpstorage ... + checkpointing run --hpc``; argparse records it as ``args.hpc`` and + ``Benchmark.write_metadata`` serializes ``vars(self.args)`` into each + run's ``metadata.json`` (so no extra plumbing is needed to surface it + to the validator). It marks an HPC shared-parallel-filesystem + environment where the §4.7.1 two-invocation failover-callout workflow + cannot complete within the 30-second budget (write and read phases are + separate scheduler jobs whose node sets cannot be guaranteed identical, + and per-invocation MPI + DLIO re-initialization over a multi-TB + checkpoint tree alone exceeds 30s). When set, ``cache_flush_validation`` + and ``checkpoint_invocation_structure`` relax their enforcement to a + warning so the exemption is visible in the report rather than silently + skipped. See Rules.md §4.7.1. + """ + for _summary, metadata, _ts in self._iter_valid_files(): + if (metadata or {}).get("args", {}).get("hpc"): + return True + return False + @rule("4.3.1", "checkpointDataSizeRatio") def checkpoint_data_size_ratio(self): """ @@ -658,6 +680,20 @@ def cache_flush_validation(self): valid = True if self.mode != "checkpointing": return valid + if self._hpc_exempt(): + # HPC exemption (--hpc): the 30-second failover-callout budget is + # infeasible on an HPC shared parallel filesystem (see _hpc_exempt). + # Downgrade to a warning so the exemption is visible in the report + # rather than enforced or silently skipped. + self.warn_violation( + "4.7.1", "checkpointCacheFlushValidation", self.path, + "HPC exemption (--hpc) declared: §4.7.1 failover-callout " + "30-second gap check skipped. On an HPC shared parallel " + "filesystem the two-invocation callout cannot meet the 30s " + "budget; cold reads are ensured by the storage layer / " + "benchmark and remain the submitter's responsibility.", + ) + return valid pairs = _pair_checkpoint_runs(self.submissions_logs) if not pairs: return valid @@ -799,6 +835,18 @@ def checkpoint_invocation_structure(self): if self.mode != "checkpointing": return valid + if self._hpc_exempt(): + # HPC exemption (--hpc): the strict 1-or-2 invocation structure is + # tied to the same-node, <=30s failover-callout workflow that is + # infeasible on an HPC shared parallel filesystem (see _hpc_exempt). + # Downgrade to a warning so the exemption is visible in the report. + self.warn_violation( + "4.7.1", "checkpointCacheFlushValidation", self.path, + "HPC exemption (--hpc) declared: §4.7.1 invocation-structure " + "enforcement relaxed for this run.", + ) + return valid + closed_runs = [ (summary, metadata, ts) for summary, metadata, ts in self._iter_valid_files() diff --git a/mlpstorage_py/tests/test_checkpoint_hpc_exemption.py b/mlpstorage_py/tests/test_checkpoint_hpc_exemption.py new file mode 100644 index 00000000..1659fdff --- /dev/null +++ b/mlpstorage_py/tests/test_checkpoint_hpc_exemption.py @@ -0,0 +1,183 @@ +"""Tests for the §4.7.1 HPC cache-flush exemption (``--hpc``). + +On an HPC shared parallel filesystem the two-invocation failover callout of +Rules.md §4.7.1 cannot meet the 30-second budget: the write and read phases are +separate batch-scheduler jobs whose node sets cannot be guaranteed identical, +and the read invocation's per-invocation MPI spawn + DLIO re-initialization over +a multi-TB checkpoint tree alone can take minutes to reach ``main.py``. + +``mlpstorage ... checkpointing run --hpc`` records ``args.hpc = True`` in each +run's ``metadata.json`` (via ``vars(self.args)``). The validator then relaxes +``cache_flush_validation`` (30s gap) and ``checkpoint_invocation_structure`` +(1-or-2 invocation shape) from errors to warnings, so the exemption is visible +in the report rather than silently applied. + +These tests lock: + * the 30s-gap check still bites a normal split run (regression guard); + * ``--hpc`` turns that failure into a pass + a warning; + * the invocation-structure check still rejects 3+ invocations normally; + * ``--hpc`` turns that failure into a pass + a warning; + * argparse registers ``--hpc`` on ``run``/``configview`` but not ``datasize``, + and it defaults False in the namespace everywhere. +""" + +from __future__ import annotations + +import argparse + +from unittest.mock import MagicMock + +from mlpstorage_py.cli.checkpointing_args import add_checkpointing_arguments +from mlpstorage_py.submission_checker.checks.checkpointing_checks import ( + CheckpointingCheck, +) +from mlpstorage_py.submission_checker.configuration.configuration import Config +from mlpstorage_py.submission_checker.loader import LoaderMetadata, SubmissionLogs + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _make_check(tmp_path, checkpoint_files): + log = MagicMock() + config = Config(version="v2.0", submitters=["Acme"], skip_output_file=True) + logs = SubmissionLogs( + checkpoint_files=checkpoint_files, + system_file=None, + loader_metadata=LoaderMetadata( + division="closed", + submitter="Acme", + system="sys-v1", + mode="checkpointing", + benchmark="llama3-70b", + folder=str(tmp_path), + ), + ) + return CheckpointingCheck(log=log, config=config, submissions_logs=logs) + + +def _split_pair(hpc): + """A two-invocation split (write 10/0, read 0/10) with a 120s gap (>30s). + + write invocation ends 05:38:00; read invocation starts 05:40:00 → 120s gap, + measured against the authoritative invocation bookends so it produces a hard + violation (not the legacy summary-origin warning) when ``--hpc`` is absent. + """ + write = ( + {"start_time": "2025-07-11T05:30:00", "end_time": "2025-07-11T05:38:00"}, + { + "verification": "closed", + "invocation_end_time": "2025-07-11T05:38:00", + "args": {"num_checkpoints_write": 10, "num_checkpoints_read": 0, "hpc": hpc}, + }, + "20250711_053000", + ) + read = ( + {"start_time": "2025-07-11T05:40:00", "end_time": "2025-07-11T05:45:00"}, + { + "verification": "closed", + "invocation_start_time": "2025-07-11T05:40:00", + "args": {"num_checkpoints_write": 0, "num_checkpoints_read": 10, "hpc": hpc}, + }, + "20250711_054000", + ) + return [write, read] + + +def _three_closed_runs(hpc): + """Three CLOSED invocations — disallowed by the invocation-structure check.""" + def run(ts, w, r): + return ( + {"start_time": "2025-07-11T05:30:00", "end_time": "2025-07-11T05:35:00"}, + { + "verification": "closed", + "args": {"num_checkpoints_write": w, "num_checkpoints_read": r, "hpc": hpc}, + }, + ts, + ) + + return [ + run("20250711_053000", 10, 0), + run("20250711_054000", 0, 10), + run("20250711_055000", 10, 10), + ] + + +def _errors_under_471(log): + return [c for c in log.error.call_args_list if "4.7.1" in str(c)] + + +# --------------------------------------------------------------------------- +# cache_flush_validation (30s gap) +# --------------------------------------------------------------------------- + +def test_split_gap_over_30s_fails_without_hpc(tmp_path): + """Regression guard: the 30s gap check still fails a normal split run.""" + check = _make_check(tmp_path, _split_pair(hpc=False)) + ok = check.cache_flush_validation() + assert ok is False + errors = [c for c in check.log.error.call_args_list if "30-second" in str(c)] + assert errors, "expected a §4.7.1 30-second gap violation without --hpc" + + +def test_split_gap_over_30s_passes_with_hpc(tmp_path): + """--hpc turns the >30s gap breach into a pass + a warning.""" + check = _make_check(tmp_path, _split_pair(hpc=True)) + ok = check.cache_flush_validation() + assert ok is True + assert _errors_under_471(check.log) == [], ( + "no §4.7.1 error should be emitted under the HPC exemption" + ) + warns = [c for c in check.log.warning.call_args_list if "HPC exemption" in str(c)] + assert warns, "the exemption must surface as a warning, not be silent" + + +# --------------------------------------------------------------------------- +# checkpoint_invocation_structure (1-or-2 invocation shape) +# --------------------------------------------------------------------------- + +def test_three_invocations_fails_without_hpc(tmp_path): + check = _make_check(tmp_path, _three_closed_runs(hpc=False)) + ok = check.checkpoint_invocation_structure() + assert ok is False + errors = [c for c in check.log.error.call_args_list if "1 or 2 invocations" in str(c)] + assert errors, "3 invocations must fail the structure check without --hpc" + + +def test_three_invocations_passes_with_hpc(tmp_path): + check = _make_check(tmp_path, _three_closed_runs(hpc=True)) + ok = check.checkpoint_invocation_structure() + assert ok is True + assert _errors_under_471(check.log) == [] + warns = [c for c in check.log.warning.call_args_list if "HPC exemption" in str(c)] + assert warns + + +# --------------------------------------------------------------------------- +# argparse registration +# --------------------------------------------------------------------------- + +def _subparser(mode, command): + parser = argparse.ArgumentParser() + add_checkpointing_arguments(parser, mode) + sub_action = next( + a for a in parser._actions if isinstance(a, argparse._SubParsersAction) + ) + return sub_action.choices[command] + + +def test_hpc_flag_registered_on_run(): + run_p = _subparser("closed", "run") + hpc = next((a for a in run_p._actions if "--hpc" in a.option_strings), None) + assert hpc is not None, "--hpc must be registered on `checkpointing run`" + assert hpc.default is False + assert hpc.const is True # store_true + + +def test_hpc_flag_absent_on_datasize_but_defaults_false(): + ds = _subparser("closed", "datasize") + opts = [s for a in ds._actions for s in a.option_strings] + assert "--hpc" not in opts, "datasize must not expose --hpc" + # set_defaults still seeds hpc=False so vars(args) always carries it. + assert ds._defaults.get("hpc") is False