reportgen silently drops kv_cache/vdb runs missing *_metadata.json — metrics publish as blank

[FileSystemGuy]

Summary

When a KVCache or VectorDB run leaf is missing its *_metadata.json file, the entire run is silently dropped from the report. The submitter's measured data is present and well-formed on disk, but every metric column for that workload comes out blank. The only trace is a WARNING line that names neither the consequence nor the missing file.

This was found while regenerating results.csv for the v3.0 submission tree: one submitter's KVCache columns published as blank even though all of their run data was intact.

Mechanism

ResultFilesExtractor.extract (mlpstorage_py/rules/models.py:976-986) resolves the benchmark type in two steps:

1. Load *_metadata.json; if present and complete, build from it. This is the only path that identifies KVCache and VectorDB runs.
2. Otherwise fall through to DLIOResultParser.parse, which infers the type from a DLIO workflow signal or Hydra configs (models.py:862-893).

A KVCache run has neither signal, so step 2 raises:

ValueError: Could not determine benchmark type for <dir>:
summary.json lacks workflow signal and no Hydra configs found

get_runs_files catches that and logs Failed to load run from {root}: {e} (mlpstorage_py/rules/utils.py:479-481), then continues. The run leaf never enters the report. Where a datasize/datagen sibling leaf still loads, the workload keeps a row in the table but with no metrics — so the output looks like a submitter with no results, not like a tooling failure.

DLIO training and checkpointing runs are unaffected: their Hydra configs still identify them, so the metadata file is effectively optional for those.

Reproducer

# Any valid kv_cache run directory
mv run/<ts>/kv_cache_<ts>_metadata.json /tmp/
mlpstorage reports reportgen --results-dir <tree>
# -> WARNING: Failed to load run from .../run/<ts>: Could not determine benchmark type
# -> results.csv: every KVCache metric column for that system is empty

Confirmed against a real submission: 5 run leaves affected, each with a well-formed summary.json (options 1/2/3 present) and all 1,836 per-rank kvcache_results_*.json files intact.

Why the current warning is not enough

The message says a run failed to load. It does not say:

• that the run's data will be omitted from the results table — the actual user-visible consequence;
• that the specific missing input is *_metadata.json;
• that this is recoverable by restoring one small file rather than rerunning the benchmark.

At scale the line is easy to miss — the run that surfaced this produced ~15,000 lines of reportgen output, in which these were 5 warnings among many.

Suggested fix

Keeping the strict behavior is defensible: arguably a submission missing *_metadata.json is incomplete, and inferring the benchmark type from summary.json shape risks masking genuinely malformed packages. But the failure should be loud and actionable. Suggestions, roughly in order of value:

1. Name the consequence and the cause. When a non-DLIO leaf fails type detection and a summary.json is present, warn along the lines of: Dropping <dir> from the report: no *_metadata.json and no DLIO workflow signal. This workload's metrics will be BLANK in results.csv. If this is a kv_cache/vector_database run, restore its <type>_<ts>_metadata.json.
2. Summarize dropped runs at the end of the reportgen pass, so the count is visible without grepping the log.
3. Optionally, have the submission checker flag a run leaf that has summary.json but no *_metadata.json, so it is caught at validation time rather than at report time.
4. Optionally, infer the type from the unmistakable KVCache summary shape (schema_version + options + trials_per_option) as a last-resort fallback that warns loudly when it fires. Listed last deliberately — the working group may prefer submitters to supply the real file.

Environment

mlpstorage_py on reportgen-column-parity; line references from that branch.

[FileSystemGuy]

Suggestions 1 and 2 are implemented on reportgen-column-parity (14987c2 RED, 2f15ab6 GREEN). Suggestions 3 and 4 are untouched — see below.

What landed

get_runs_files takes an optional skipped list and records every directory it could not load as a SkippedRunDir(path, reason, detail). Three reasons are distinguished:

reason	meaning
undetermined_type	no usable *_metadata.json and no DLIO workflow signal — the case this issue describes
multiple_metadata	more than one *_metadata.json in a leaf
load_error	anything else

The first is separated from the rest by a new BenchmarkTypeUndetermined exception rather than by matching the message text. It subclasses ValueError, so existing handlers are unaffected.

Suggestion 1 — the warning now names the consequence and the remedy:

Dropping <dir> from the report: no usable *_metadata.json and no DLIO
workflow signal. This workload's metrics will be BLANK in results.csv.
If this is a kv_cache or vector_database run, restore its
<benchmark-type>_<run-datetime>_metadata.json file — the measured data
does not need to be regenerated.

Suggestion 2 — ReportGenerator collects drops across all scan roots, logs a count mid-pass, and prints a SKIPPED RUN DIRECTORIES section at the end of the report. That section also prints on the all-dropped path, where print_results short-circuits to "No results to display" and previously said nothing at all about what it had thrown away — which is exactly when the information matters most.

Two things not in the original report

Incomplete metadata fails identically to absent metadata. ResultFilesExtractor.extract gates on metadata and self._is_complete_metadata(metadata), so a metadata file present but missing one of benchmark_type / run_datetime / num_processes / parameters falls through to the same DLIO parser and the same silent drop. Both cases are now reported and both are test-pinned.

The #836 warning mechanism cannot cover this. Those warnings attach to Result.issues, but a dropped leaf never becomes a Result — the drop happens up in get_runs_files. This needed a separate channel, not a reuse of what #836 added.

Verification

Against a hardlink clone of the v3.0 submission tree (the real tree was not written to):

• zero directories are being skipped as it stands today. The 5 leaves that surfaced this issue were repaired by the metadata restoration in submissions PR fix workload path in README #32, and nothing else in the tree hits any of the three drop paths.
• Positive control with one Everpure kv_cache and one TTA vector_database metadata file moved aside: both are named in the section with the correct count and the correct remedy.

14 new tests in tests/unit/test_skipped_run_visibility.py, including negative controls for a clean tree and for callers that pass no skipped list. Seven get_runs_files test doubles in test_reporting.py pinned the old two-argument signature; because accumulate_results wraps the call in except Exception, the resulting TypeError surfaced as an empty scan rather than an error. Those doubles now take **kwargs.

Still open

Suggestion 3 (submission checker flags a run leaf with summary.json but no *_metadata.json) and suggestion 4 (infer the type from the KVCache summary shape as a last-resort fallback) are both unimplemented. Suggestion 4 in particular is a policy call for the working group — whether submitters must supply the real file — and shouldn't be decided by the reporting layer alone.

Related: the structural layer had the same class of problem, where a model directory the walker cannot descend produced only No valid run directories found in <dir> and stopped, leaving everything beneath it unexamined and unmentioned. That is addressed separately on the same branch (ba426d3, e0e74bc).