checkpointing: --hpc flag to relax §4.7.1 cache-flush gap on HPC shared filesystems

ManPage.md

@@ -607,6 +607,9 @@ Required positionals: `<model>` (one of `llama3-8b`, `llama3-70b`, `llama3-405b`
 - **`--checkpoint-folder <path>`, `-cf <path>`**
   Storage location for checkpoint files. Required for `run`.
 
+- **`--hpc`** *(run/configview only)*
+  Declare an HPC shared-parallel-filesystem environment where the §4.7.1 two-invocation failover callout cannot meet the 30-second budget (the write and read phases are separate scheduler jobs whose node sets cannot be guaranteed identical, and per-invocation MPI + DLIO re-init over a multi-TB tree alone exceeds 30s). Relaxes the §4.7.1 30-second gap check and the 1-or-2 invocation-structure check to warnings, and is recorded in `metadata.json` for auditability. Cold reads must still be ensured by the storage layer / benchmark (remote parallel FS, `--o-direct`, or `posix_fadvise(DONTNEED)` on close). See Rules §4.7.1.
+
 - **`--hosts`, `--exec-type`, `--dlio-bin-path`**
   Same semantics as the training options of the same name.
 

Rules.md

@@ -520,7 +520,7 @@ root_folder (or any name you prefer)
 
 ## 4.7. Storage System Must Be Simultaneously R/W or _Remappable_
 
-4.7.1. **checkpointCacheFlushValidation** -- Checkpointing models the failure of a client node followed by another client picking up the last checkpoint file written by the failed node for the read phase. In every submission the write phase (10 checkpoint files written) runs first, followed by the read phase (10 checkpoint files read). When the storage system supports the client-to-client handoff transparently — i.e., the read phase can proceed immediately after the write phase without external orchestration — the write and read phases may be executed as a single combined invocation, and no gap check applies. Storage system architectures that require an external callout (e.g., a submitter-provided script) to complete the failover between the writing and reading clients must instead execute the write and read phases as two separate invocations, with the submitter's failover callout occurring between them. A common in-callout activity is clearing a client-side filesystem cache when the total checkpoint size written per client is less than 3× the client node's memory capacity (see ``checkpointing/README.md``); the callout is not limited to that activity. To ensure the callout is a lightweight programmatic step rather than a long-running manual procedure, the validator confirms that the read-phase invocation was launched — i.e., its ``mlpstorage`` process reached the entry point of ``main.py`` — within 30 seconds of the write-phase invocation ending. The gap is measured as ``read.invocation_start_time − write.summary.end_time``, where ``invocation_start_time`` is captured at ``mlpstorage`` process start (before framework startup, MPI spawn, and other unavoidable per-invocation overhead) and ``end_time`` is recorded in the write invocation's ``summary.json``. A negative gap indicates clock skew between the write and read nodes and is reported as such rather than as a causality violation.
+4.7.1. **checkpointCacheFlushValidation** -- Checkpointing models the failure of a client node followed by another client picking up the last checkpoint file written by the failed node for the read phase. In every submission the write phase (10 checkpoint files written) runs first, followed by the read phase (10 checkpoint files read). When the storage system supports the client-to-client handoff transparently — i.e., the read phase can proceed immediately after the write phase without external orchestration — the write and read phases may be executed as a single combined invocation, and no gap check applies. Storage system architectures that require an external callout (e.g., a submitter-provided script) to complete the failover between the writing and reading clients must instead execute the write and read phases as two separate invocations, with the submitter's failover callout occurring between them. A common in-callout activity is clearing a client-side filesystem cache when the total checkpoint size written per client is less than 3× the client node's memory capacity (see ``checkpointing/README.md``); the callout is not limited to that activity. To ensure the callout is a lightweight programmatic step rather than a long-running manual procedure, the validator confirms that the read-phase invocation was launched — i.e., its ``mlpstorage`` process reached the entry point of ``main.py`` — within 30 seconds of the write-phase invocation ending. The gap is measured as ``read.invocation_start_time − write.summary.end_time``, where ``invocation_start_time`` is captured at ``mlpstorage`` process start (before framework startup, MPI spawn, and other unavoidable per-invocation overhead) and ``end_time`` is recorded in the write invocation's ``summary.json``. A negative gap indicates clock skew between the write and read nodes and is reported as such rather than as a causality violation.  **HPC exemption:** on an HPC shared parallel filesystem, the two-invocation failover callout cannot meet the 30-second budget — the write and read phases are separate batch-scheduler jobs whose node sets cannot be guaranteed identical, and the read invocation's unavoidable per-invocation MPI spawn plus DLIO re-initialization over a multi-TB checkpoint tree alone can take several minutes to reach ``main.py``.  A submitter may declare this environment by passing ``--hpc`` to ``mlpstorage ... checkpointing run``.  When ``--hpc`` is recorded in the run's ``metadata.json``, the validator relaxes the 30-second gap check and the 1-or-2 invocation-structure check to warnings (so the exemption is visible in the report rather than silently applied).  On such systems cold reads are ensured by the storage layer and the benchmark itself — a remote parallel filesystem, ``O_DIRECT`` (``--o-direct``), and/or the benchmark's ``posix_fadvise(POSIX_FADV_DONTNEED)`` on file close — rather than by the callout timing; the submitter remains responsible for ensuring the read phase is served cold from storage.
 
 4.7.2. **checkpointTotalTestDuration** -- The validator must verify that the total test duration starts from the timestamp of the first checkpoint written and ends at the ending timestamp of the last checkpoint read, notably including the "remapping" time.
 

mlpstorage_py/cli/checkpointing_args.py

@@ -57,13 +57,16 @@ def _add_checkpointing_core_args(parser, command):
         parser: The subcommand parser to add arguments to.
         command: The subcommand name ('datasize', 'run', 'configview').
     """
-    # Set defaults for open-gated attrs so they always exist in the namespace
+    # Set defaults for open-gated attrs so they always exist in the namespace.
+    # hpc defaults False here so vars(args) always carries it (recorded in
+    # metadata.json) even on subcommands that don't expose the flag.
     parser.set_defaults(
         loops=1,
         params='',
         allow_invalid_params=False,
         dlio_bin_path=None,
         checkpoint_folder=None,
+        hpc=False,
     )
 
     add_host_arguments(parser)
@@ -137,6 +140,32 @@ def _add_checkpointing_core_args(parser, command):
             ),
         )
 
+    # --hpc: available for run and configview (not datasize).
+    # Declares an HPC shared-parallel-filesystem environment where the §4.7.1
+    # two-invocation failover callout cannot complete within the 30-second
+    # budget (the write and read phases are separate scheduler jobs on nodes
+    # that cannot be guaranteed identical, and per-invocation MPI + DLIO
+    # re-initialization over a multi-TB checkpoint tree alone exceeds 30s).
+    # See mlcommons/storage and Rules.md §4.7.1.
+    if command in ("run", "configview"):
+        parser.add_argument(
+            '--hpc',
+            action='store_true',
+            default=False,
+            dest='hpc',
+            help=(
+                "Declare an HPC shared-parallel-filesystem environment where "
+                "the §4.7.1 same-node, <=30s two-invocation failover-callout "
+                "workflow is infeasible. Relaxes the 30s inter-phase gap and "
+                "invocation-structure enforcement for this run. Recorded in "
+                "metadata.json for auditability; on such systems cold reads "
+                "are ensured by the storage layer / benchmark (e.g. "
+                "posix_fadvise(DONTNEED) on close, O_DIRECT, or a remote "
+                "parallel filesystem) rather than by the callout timing, and "
+                "the submitter remains responsible for cold reads."
+            ),
+        )
+
     # Checkpoint folder required for run only
     if command == "run":
         parser.add_argument(

mlpstorage_py/cli/help_formatter.py

@@ -296,11 +296,17 @@
     --hosts/-s HOST...              (default: 127.0.0.1)
     --num-checkpoints-read/-ncr N   (default: 10; closed allows 10 or 0)
     --num-checkpoints-write/-ncw N  (default: 10; closed allows 10 or 0)
+    --hpc                           (relax §4.7.1 30s gap + invocation
+                                     structure for HPC shared filesystems)
   + MPI_ARGS
   + CORE_STD
   Note: closed runs use 10/10 by default. Use 10/0 then 0/10 in two
         invocations when a cache flush is required between phases
         (see Rules.md §4.7.1 and checkpointing/README.md).
+  Note: --hpc declares an HPC shared-parallel-filesystem environment where
+        the two-invocation failover callout cannot meet the 30s budget; it
+        relaxes the §4.7.1 gap + invocation-structure checks to warnings and
+        is recorded in metadata.json (see Rules.md §4.7.1).
 
   Closed rank constraints by model:
     llama3-1t:   8 or 1024

mlpstorage_py/submission_checker/checks/checkpointing_checks.py

@@ -114,7 +114,29 @@ def _iter_valid_files(self):
                 )
                 continue
             yield summary, metadata, timestamp
-
+
+    def _hpc_exempt(self):
+        """True if any run in this workload declared the §4.7.1 HPC exemption.
+
+        The exemption is opted into at run time with ``mlpstorage ...
+        checkpointing run --hpc``; argparse records it as ``args.hpc`` and
+        ``Benchmark.write_metadata`` serializes ``vars(self.args)`` into each
+        run's ``metadata.json`` (so no extra plumbing is needed to surface it
+        to the validator). It marks an HPC shared-parallel-filesystem
+        environment where the §4.7.1 two-invocation failover-callout workflow
+        cannot complete within the 30-second budget (write and read phases are
+        separate scheduler jobs whose node sets cannot be guaranteed identical,
+        and per-invocation MPI + DLIO re-initialization over a multi-TB
+        checkpoint tree alone exceeds 30s). When set, ``cache_flush_validation``
+        and ``checkpoint_invocation_structure`` relax their enforcement to a
+        warning so the exemption is visible in the report rather than silently
+        skipped. See Rules.md §4.7.1.
+        """
+        for _summary, metadata, _ts in self._iter_valid_files():
+            if (metadata or {}).get("args", {}).get("hpc"):
+                return True
+        return False
+
     @rule("4.3.1", "checkpointDataSizeRatio")
     def checkpoint_data_size_ratio(self):
         """
@@ -658,6 +680,20 @@ def cache_flush_validation(self):
         valid = True
         if self.mode != "checkpointing":
             return valid
+        if self._hpc_exempt():
+            # HPC exemption (--hpc): the 30-second failover-callout budget is
+            # infeasible on an HPC shared parallel filesystem (see _hpc_exempt).
+            # Downgrade to a warning so the exemption is visible in the report
+            # rather than enforced or silently skipped.
+            self.warn_violation(
+                "4.7.1", "checkpointCacheFlushValidation", self.path,
+                "HPC exemption (--hpc) declared: §4.7.1 failover-callout "
+                "30-second gap check skipped. On an HPC shared parallel "
+                "filesystem the two-invocation callout cannot meet the 30s "
+                "budget; cold reads are ensured by the storage layer / "
+                "benchmark and remain the submitter's responsibility.",
+            )
+            return valid
         pairs = _pair_checkpoint_runs(self.submissions_logs)
         if not pairs:
             return valid
@@ -799,6 +835,18 @@ def checkpoint_invocation_structure(self):
         if self.mode != "checkpointing":
             return valid
 
+        if self._hpc_exempt():
+            # HPC exemption (--hpc): the strict 1-or-2 invocation structure is
+            # tied to the same-node, <=30s failover-callout workflow that is
+            # infeasible on an HPC shared parallel filesystem (see _hpc_exempt).
+            # Downgrade to a warning so the exemption is visible in the report.
+            self.warn_violation(
+                "4.7.1", "checkpointCacheFlushValidation", self.path,
+                "HPC exemption (--hpc) declared: §4.7.1 invocation-structure "
+                "enforcement relaxed for this run.",
+            )
+            return valid
+
         closed_runs = [
             (summary, metadata, ts)
             for summary, metadata, ts in self._iter_valid_files()