Conversation
…n, lib refactor map
* feat(dtni): vllm_single PoC — typed configs + orch-driven VllmJob
Replace 4 byte-similar vllm_single wrappers with a single parametrized
suite, per-variant config + threshold dirs, a typed pydantic loader, and
a new orch-driven VllmJob whose container lifecycle is owned entirely by
ContainerOrchestrator (launch:true).
- cvs/lib/dtni/{verdict,config_loader}.py — 5 threshold kinds; pydantic v2
models with extra=forbid; 3-pass placeholder substitution; model.remote=1
raises NotImplementedError pointing at v1 resource_resolver.
- cvs/lib/inference/vllm_orch.py — standalone VllmJob driven by orch.exec.
Drops dead self.port_no distributed branch, random_range_ration typo,
globals.error_list indirection, silent-skip in verify_inference_results.
- cvs/tests/inference/vllm/{conftest,_shared,vllm_single}.py — orch fixture
owns container lifetime; test_print_results_table moved to _shared.
- cvs/input/dtni/vllm_single/{4 variants}/{config,threshold}.json — all
variants pinned to rocm/vllm-dev:nightly for the PoC; thresholds carry
MI300X-realistic floors (~1/3 of MI355X totals) for the verification node.
- cvs/input/cluster_file/mi300x_g21u37.json — single-node MI300X cluster
for verification on 10.245.135.13.
- Delete the 4 old per-model wrappers and mi355x_vllm_single.json.
Verification (offline gates): pytest --collect-only enumerates 9 parametric
cells + test_print_results_table; cvs list vllm_single discovers both test
functions; load_variant of a missing-models-dir variant resolves the
expected /models/{id} path; pydantic ValidationError fires on a
percentile_metrics typo. On-hardware verification is deferred: target node
lacks pre-fetched models, HF token, and benchmark server scripts.
Legacy cvs/lib/inference/{base,vllm,inference_max}.py untouched; other
suites (sglang, inferencemax, pytorch_xdit, megatron, jax) unaffected.
* fix(dtni): cluster file uses devbox-correct key path
The devbox /data/atnair is /data/atnair (not /home/atnair), and the node
10.245.135.13 authenticates with id_ed25519 (not id_rsa). Update the
verification cluster file so the orch fixture authenticates first try.
Confirmed via a lifecycle smoke that brought up an alpine container on
the node and tore it down at the right boundaries.
* fix(dtni): remediate vllm_single lifecycle review findings
- is_ready: grep in-container instead of cat-ing the whole server log
- thresholds: fail at load if sweep cells lack a threshold entry; hard-error
(not silent skip) on per-cell verdict miss
- HTML report: per-test timing rows with explicit units (no cross-row leak)
- client failure: treat only a nonzero failed-request count as failure
- pin HF cache to the mounted models dir; shlex-quote shell interpolation
- raise server readiness budget to 60min for remote model pulls
- remove legacy cvs.lib.inference.vllm.VllmJob (no remaining importers)
* refactor(dtni): collapse vllm_single to single W1 config + generic cluster file
- Replace 4 model variants under input/dtni/vllm_single/ with one W1 config
(Llama 3.1 70B FP8-KV, TP=8) at input/config_file/inference/vllm_single/.
- Rename cluster file to mi300x_vllm_single.json with <changeme> placeholders
so it is generic/shareable rather than node-specific.
- config_loader: add enforce_thresholds gate (record-only scaffolds),
glob threshold sibling, drop enumerate_variants, note generalization seam.
- Move pytest_generate_tests into vllm_single test module; drop aa/ab/zz
lifecycle-ordering prefixes from test names.
- Drop unused imports / apply ruff formatting across touched lib + test files.
* fix(dtni): address vllm_single PR review findings
- config_loader: drop dead BenchmarkParams class + unused benchmark_params
field (and the matching key in the w1 config)
- config_loader: raise FileNotFoundError/ValueError instead of AssertionError
in load_variant (AssertionError is stripped under python -O)
- config_loader: collapse _resolve_cluster_mapping; clarify the container
runtime docstring and the intentional model_validator ordering
- vllm_orch: build the bench client command as a shlex.quote-d arg list so a
model id or path containing a space or $ cannot break the inner bash layer
* fix(dtni): address second-round vllm_single review
- conftest: scope inf_res_dict per-module to match sibling fixtures and
avoid cross-module result-table bleed
- test_model_fetch: split the offline/pre-staged path from the download
poll loop; presence-check with retries so a slow mount that reads 0 on
the first du does not false-fail a model that is present
- start_server: shlex.quote scripts_dir/server_script/server_log, matching
the per-path quoting used elsewhere in the file
- test_teardown: set lifecycle.torn_down only after verifying the container
is gone so the orch finalizer retries an incomplete teardown
- _clone_bench_serving: document why the bench_serving URL is a hardcoded
calibration fork (not stock vLLM, unpinned HEAD), kept for legacy parity
…#227) * feat(dtni): move vllm bench client to stock vllm bench serve Drop the kimbochen/bench_serving fork clone in VllmJob.run_client and invoke the in-image stock "vllm bench serve" CLI instead. The fork was cloned at unpinned HEAD; pinning the client to the run image tag gives Spec 1 a stable artifact contract to parse. - run_client: remove _clone_bench_serving call; head tokens become vllm/bench/serve; drop the now-dead cd /app in client_cmd - remove _clone_bench_serving and its (false) calibration comment - drop Params.bench_serv_script (extra=forbid) and the matching key in the vllm_single config; rename client_log to client.log base.py / inferencemax still use the fork (separate workload, untouched). Source-only change; no metrics added (Spec 1). enforce_thresholds=false. * fix(dtni): robust client completion + launch-failure detection for stock bench Harden wait_client_complete after the move to stock vllm bench serve: - COMPLETION_RE: key off the unconditional "Serving Benchmark Result" banner instead of the "End-to-end Latency" metric header. Stock prints metric headers only when the metric is in --percentile-metrics, so a config omitting e2el would never be detected as complete and would spin to the poll cap (~90 min) on an otherwise-successful run. - add CLIENT_LAUNCH_FAIL_RE: a CLI launch failure (bad/renamed flag, missing bench subcommand, vllm not on PATH) exits before any summary and is neither a Python traceback nor a Failed-requests line, so it too would hang the poll cap. Treat it as a hard failure, like a crash. - drop dead export RESULT_FILENAME=results: consumed only by the removed fork client; stock takes the name via --result-filename.
Hnimrama/inferencemax uplift Refactors InferenceMax for the DTNI pytest layout (inferencemax_single): ContainerOrchestrator-based conftest, suite/threshold JSON loading, benchmark model selection, and tighter server/client lifecycle handling against current InferenceX upstream. Benchmarking: stop cloning third-party bench_serving; resolve benchmark_serving.py from the installed vllm package (BENCH_SCRIPT) for InferenceMax and vLLM single paths. Host-mounted server entrypoints live under cvs.lib.dtni.vllm_benchmark_scripts (vllm_serve_mi300x.sh); samples and docs use <changeme> container placeholders, legacy benchmark_script_repo called out as ignored, and volume_dict guidance avoids duplicate Docker :/workspace mounts. vLLM single (vllm_orch): align with dev/dtni completion and client-failure detection while keeping python3 "$BENCH_SCRIPT" invocation. Misc: optional run_plugin --log-file; sglang_disagg total_generated_tokens key; log redaction and small review fixes from PR feedback. Test with cvs run inferencemax_single (cluster + suite JSON, HF token) and spot-check vllm_single if configs touch shared modules.
…llm result path (#233) * refactor(lib): rename dtni -> utils and split out generic config machinery Rename cvs/lib/dtni to cvs/lib/utils ("utils" says what it is: pure functions any lib can call; "dtni" was a leftover project codename). verdict.py moves unchanged. config_loader.py is trimmed to the framework-agnostic half: the paths/model/image/container schema, the 3-pass placeholder substitution, the enforce_thresholds gate on a new BaseVariantConfig, and a substitute_config() helper (file read + substitution + sibling-threshold discovery). The inference-only schema moves to a sibling module in a later commit. * refactor(inference): move vllm_parsing into cvs/lib/inference/utils The client.* metric parser (to_client_metrics + CLIENT_METRICS surface) is inference-specific and should not sit in the shared utils dir. Move it under a new cvs/lib/inference/utils package. Content unchanged. * feat(inference): inference config schema with named-combo sweep selector The inference half of the old config_loader: GoodputSlo, SeqCombo, Sweep, Params, Roles, and VariantConfig(BaseVariantConfig) with cell_key and the threshold-coverage check. load_variant() delegates the file read and placeholder substitution to the generic substitute_config(). Replaces the sequence_combinations x concurrency_levels cartesian with a named-combo + explicit runs[] selector: each run is a {combo, concurrency} pair, so the config enumerates exactly the cells to run (no NxM explosion). A model_validator rejects duplicate combo names and runs referencing an unknown combo at load time. * feat(inference): self-contained server cmd + artifact-based result parsing build_server_cmd/start_server assemble a `vllm serve` arg list in Python (mirroring run_client) instead of cloning and running an external .sh, so a run needs no hand-staged script. --max-model-len is derived per cell from isl/osl/random_range_ratio so any sweep change stays self-consistent. parse_results reads the stock extensionless `results` JSON artifact that `vllm bench serve` writes to --result-dir and delegates namespacing + derived-metric math to vllm_parsing.to_client_metrics, replacing the brittle console-log regex table. Missing/empty/unparseable artifacts hard-fail the cell rather than recording a silently-green empty row. Adds the optional --goodput SLO gate (per-cell, omitted when no SLO) and threads goodput_slo through run(). * feat(suite): per-metric result rows, Value/Unit columns, sweep selector pytest_generate_tests now drives parametrization from the named-combo + runs[] selector instead of the cartesian. test_vllm_inference only runs the benchmark and stashes results; the verdict moves into a new test_metric (one pytest test = one HTML row per metric per cell), with inline Value/Unit columns added via pytest_html hooks in conftest. test_setup_sshd gates its 2224 probe on len(orch.hosts) > 1, mirroring the single-node orchestrator guard: single-node runs skip the in-container sshd (it exists only for inter-node MPI) and must not probe for it. Import paths follow the dtni -> utils / inference.utils moves. * chore(config): rename vllm_single config pair, adopt selector, drop cluster file Rename the config/threshold pair to {model}_{precision}_{config|threshold} .json and convert the sweep to the named-combo + runs[] selector. Pin the image to rocm/vllm-dev:nightly (the previously pinned :nightly-sshd tag does not exist on Docker Hub, and single-node runs skip in-container sshd). Delete cvs/input/cluster_file/mi300x_vllm_single.json: a cluster file only needs node IP + user/key/orchestrator; the variant config supplies the container block, so the bespoke per-suite cluster file is redundant. * test(inference): unit tests for parser, sweep selector, and verdict guards Cover to_client_metrics purity + derived metrics, the named-combo/runs sweep selector (expansion, unknown-combo and duplicate-name rejection), the run_client goodput/metric-percentiles flags, table-cell rendering, and the verdict None-guards. Adds JSON fixtures for the stock results artifact. * docs: suite-authoring guide + AGENTS.md for shared and inference helpers Add a human reference guide (plans/building-a-cvs-test-suite.md) that walks the six-layer suite architecture using vllm_single as the worked example: the generic <-> framework config seam, the named-combo + runs[] sweep selector, the self-contained Python-built server cmd, lifecycle-as-tests, and a checklist for authoring a new inference or training suite. Add per-package AGENTS.md docs naming the public entry points, the seam, and the non-obvious gotchas: - cvs/lib/utils: substitute_config / BaseVariantConfig / evaluate_all, the 3-pass placeholder order, sibling-glob threshold discovery, parent-first validator ordering. - cvs/lib/inference/utils: load_variant / to_client_metrics / CLIENT_METRICS, the cell_key single-source-of-truth, the coverage check that prevents a silent green, and the validators mirrored in pytest_generate_tests. * docs: expand suite guide with lib restructure, drop redundant section rules Document the dtni -> utils rename and the shared (cvs/lib/utils) vs domain-specific (cvs/lib/inference/utils) split: what lives where, the rule for placing a new helper, and the directory map. Note training is not yet ported and this guide is the blueprint for that port. Remove the manual --- horizontal rules between sections: heading levels already render their own bottom border, so the extra rules produced a double-underline. Minor prose/format cleanups. * docs: demote headings so GitHub stops underlining sections * removing old plan * fix(config): re-key W1 threshold to the swept CONC=16 cell The threshold file carried placeholder CONC=64/128/256 entries while the sweep's only run is concurrency 16, so cell_key() matched no threshold. The mismatch was masked by enforce_thresholds=false (warned, not raised) and would have failed load the instant enforcement was flipped on. Re-key to the single CONC=16 cell the runs selector actually enumerates. * fix(inference): drop dead server-env exports from build_server_cmd MODEL/ISL/OSL/MAX_MODEL_LEN/RANDOM_RANGE_RATIO/TP/CONC/PORT were exported into /tmp/server_env_script.sh but read by nothing after the .sh->Python server command refactor -- both _server_argv and run_client pass these as explicit flags. Keep only the env the vllm process actually consumes (HF token, HF cache pin, AITER flags). Also drops the second _derive_max_model_len call that fed the dead MAX_MODEL_LEN export. * fix(inference): move --kv-cache-dtype out of the driver into config _server_argv hard-coded --kv-cache-dtype fp8, baking a per-model property into the shared orchestrator -- a non-fp8-KV model dropped into the suite would be served wrong with no config recourse (extra_serve_args can only add, so an override would pass the flag twice). Declare it in the W1 config's roles.server.extra_serve_args instead; the driver stays model-agnostic and 'new model = new config' holds. * refactor(inference): share the sweep-selector validator across load and collection pytest_generate_tests hand-reimplemented the duplicate-name and unknown-run.combo checks that Sweep._check_runs_reference_known_combos already enforces, with divergent semantics (first-failure raise vs all-at-once). Extract validate_sweep_selector() as the single home and call it from both the typed validator (load time) and the collection-time raw-JSON path so the rule can't drift. * fix(inference): key the per-cell out_dir by isl/osl/conc out_dir was fixed per job, so a multi-cell sweep would overwrite each cell's `results` and client.log, and parse_results could cat a prior cell's stale artifact if the current cell's client failed to write one. Key it by cell. Latent today (the shipped sweep has one cell). * refactor(inference): normalize goodput_slo to dict-only run_client accepted goodput_slo as either a dict (.get) or an object (getattr) via a per-key hasattr branch, but the only production caller passes a raw dict -- the object path existed solely for a unit test, and the dual path meant the typed GoodputSlo's validation never reached the command builder. Consume the dict only and drop the object-form test. * test(inference): drop unused _fake_variant parameter goodput_slo_unused was never read (goodput is threaded through _make_job). * chore(inference): placeholder personal/image refs in example config The committed vllm_single example config carried a personal hf-token path and a concrete image tag (duplicated in image.tag and container.image). Replace all three with <changeme> so the file is a template a new user must fill in -- it still loads (collection works) and only fails at run time when an unedited value is read, which is the intended signal. * refactor(inference): server serve_args as a {flag: value} map The per-model server knobs were a flat [flag, value, flag, value] list (roles.server.extra_serve_args), which reads poorly. Replace with a roles.server.serve_args {flag: value} map (flag without the leading --): a scalar renders --flag value, True a bare --flag, a list the flag once per element -- so it stays readable while still covering vllm bare/repeatable flags. _server_argv flattens the map via a new _flatten_serve_args helper; the derived flags (tp/max-model-len/port) stay code-built. Also repoints a stale unit test that asserted MAX_MODEL_LEN in the env script (it moved to the --max-model-len flag in an earlier commit) to assert against the server argv instead. * feat(verdict): add unit-agnostic max threshold kind The only ceiling kind was max_ms, whose message hard-codes ms. A count metric like client.failed needs an upper bound without the unit lie; add a plain max with the same comparison and an honest message. * feat(inference): enforce a declared gated-metric SLO contract Previously only cell-presence was validated: a cell could exist while a given metric had no spec, and test_metric (spec is None -> return) would silently report a green record-only row even under enforce_thresholds=true. A new perf metric was thus unvalidated by default. Declare GATED_METRICS beside CLIENT_METRICS -- the perf+health subset that must assert (throughput, mean+p99 latency, success_rate/failed) -- and extend _check_thresholds_cover_sweep to require a spec for every gated metric in every present cell, reusing the same enforce-vs-warn path. A new metric is record-only until added to the set; once gated, the loader forces a spec in every cell before the suite can run green. Inputs, totals, and derived diagnostics stay record-only by design. * fix(inference): single image source on container.image The image was declared twice: top-level image.tag (live -- conftest copied it onto the container block) and container.image (dead -- overwritten by that copy). The duplicate forced a top-level image block whose remote field was unused and whose tag silently shadowed container.image. Make container.image the single source: drop the top-level ImageSpec block from the generic BaseVariantConfig, drop the conftest overwrite so the merged container.image is used as-is, and remove the now-schemaless image block from the example config. * feat(inference): gate the full latency distribution Expand GATED_METRICS from the mean+p99 subset to every emitted latency quantile (mean/median/p90/p95/p99) for ttft, tpot, itl, and e2el -- itl omits p90 as CLIENT_METRICS has no producer for it. Throughput and success_rate/failed health are unchanged. Inputs, totals, secondary throughputs, and derived diagnostics stay record-only. The example threshold file gains a placeholder spec for each newly gated metric (23 total) so the loader gated-coverage check passes. * docs: reflect image collapse, serve_args rename, max kind, GATED_METRICS - utils/AGENTS.md: drop top-level image (now container.image); add max verdict kind - inference/utils/AGENTS.md: document GATED_METRICS contract + dual-axis coverage check - building-a-cvs-test-suite.md: container.image, serve_args map, max kind, GATED_METRICS - dtni-dev-guide.md: SUPERSEDED banner pointing to building guide + AGENTS.md * refactor(inference): rename vllm_orch → vllm_single throughout The module name vllm_orch.py implied disaggregated orchestration; this is a single-node suite. Align the module name with the suite file and suite name. Update all import sites, AGENTS.md prose, and the plan doc. * fix(config): drop ModelSpec.precision, make threshold_json an explicit field ModelSpec.precision was an unvalidated free-text field with no downstream use; the kv-cache-dtype flag belongs in serve_args. Removing it prevents configs silently carrying a stale or misleading label. Replace the sibling-glob threshold discovery (glob('*threshold.json') next to the config) with an explicit threshold_json field on BaseVariantConfig. The glob was fragile: ambiguous when multiple threshold files coexist, and invisible in the config spec. An explicit absolute path is transparent, repo-portable, and validated as part of the schema. Update the example config to add threshold_json: "<changeme>" and drop model.precision. * test(config): unit tests for ModelSpec, BaseVariantConfig, substitute_config Cover the contracts changed by the precision-removal and threshold_json-explicit commits: ModelSpec forbids precision and extra keys; BaseVariantConfig requires threshold_json; substitute_config reads the threshold via raw['threshold_json'] as a literal absolute path, not by globbing the sibling directory; a sibling *threshold.json must NOT be auto-discovered (regression guard for old behavior); comment keys are stripped. No hardware; pure filesystem via tempfile. * refactor(inference): O(n) duplicate detection in validate_sweep_selector Replace list.count() inside the loop (O(n²)) with Counter: one pass to build the frequency map, one comprehension to collect duplicates. Suggested in review. * test(inference): address review — setUpClass, if __name__ at end of file Convert per-test module loads to setUpClass so each heavy import runs once per class, not once per test method: TestTableCellRendering._cell() (loads _shared.py + stubs tabulate), TestKeyConsistency._producer_keys() (runs parse_results), TestMetricTests.setUp() (_load_vllm_single). Suggested in review for TestTableCellRendering; applied the same fix to the other two classes that had the identical problem. Move `if __name__ == "__main__": unittest.main()` from mid-file (line 282) to the very end. Test classes defined after the guard were still discovered by both pytest and python -m unittest (Python parses the full file first), but the placement looked like dead code. Flagged in review. * test(inference): address review + expand coverage for config loader models Review items: remove dead `vc = _variant(sw)` assignment that was never read; move `import warnings` from inline to top-level imports. New test classes covering contracts changed in this PR: TestModelSpecNoPrecision (extra field rejected), TestThresholdJsonField (required field, constructs with it), TestCellCoverageAxis (missing cell and extra threshold-key axes, warn/raise modes), TestExpectedCellsBoundaries (empty runs, unreferenced combo), TestGoodputSlo (construction, missing fields, forbid extra keys, optional on SeqCombo), TestSeqComboForbid (required fields, extra keys). All no-hardware. * feat(config): expand w1 llama31_70b_fp8kv sweep to 5 ISL/OSL cells at conc=16 Replace the single ISL=128/OSL=2048 placeholder cell with the full ISL/OSL matrix requested in review: ISL=1024 / OSL=1024 ISL=8192 / OSL=1024 ISL=1K / OSL=8192 ISL=1K / OSL=4096 ISL=5000 / OSL=1024 All five cells run at concurrency=16, TP=8, with record-only placeholder thresholds (enforce_thresholds=false). threshold.json carries a spec for every GATED_METRICS member per cell so the loader coverage check passes. * fix(config): use literal ISL/OSL values (1000/8000/4000 not 1024/8192/4096) * style(inference): collapse multi-line string concat in launch cmd Co-Authored-By: Claude <noreply@anthropic.com> * adding documentation * renaming config and threshold files --------- Co-authored-by: Claude <noreply@anthropic.com>
Squashed from 74 incremental commits merged via PR #234: SGLang multinode orchestration, smoke tests, hellaswag accuracy runs, and iterative framework updates. Full per-commit history preserved in the backup branch/tag solaiys/dev-dtni-presquash-2026-08-02.
…epseek r1 model * Restore InferenceMax uplift reverted by #228 Reverts commit 4a8425f, restoring the changes from PR #225 on dev/dtni. * fix(inference): run vLLM bench client with vLLM interpreter Probe python3.13..python3 for import vllm; export BENCH_PY and BENCH_SCRIPT. Use shlex.quote for docker exec bash -c. Align InferenceMax client completion with Serving Benchmark Result or End-to-end Latency. * fix(dtni): broaden vLLM benchmark script discovery Search site-packages and ancestor paths, verify the file is readable, and document vllm[bench] when wheels omit benchmarks/. * fix(inference): harden InferenceMax server startup and GPU mem env Use CVS_GPU_MEMORY_UTIL in sample config and serve script to avoid vLLM unknown-env warnings. Extend default readiness poll budget to 60 and grep full server logs so Uvicorn ready is not missed after long model loads. * fix(dtni): fall back to vllm bench serve when benchmark script is absent Wheels often omit vllm/benchmarks; resolve the driver via eval exports, run python -m vllm.entrypoints.cli.main bench serve when needed, and fail fast on missing-script log patterns in InferenceMax and base polling. * fix(dtni): clamp bench random-range to max_model_length vLLM random workloads scale (ISL+OSL)*(1+r); clamp ratio when it would exceed MML, pass --temperature 0 for greedy parity, and forward --metric-percentiles in InferenceMax and vllm_single clients. * fix(inference): extend InferenceMax bench client poll budget Read client_poll_count and client_poll_wait_time from benchmark_params (defaults 50/60), document them and fix the inferencemax.rst table, and surface the keys in sample MI300X/MI355X configs. * feat(inference): add bench_max_failed_requests cap and completion-first polling Gate benchmark success on Failed requests only after the summary is present; tail more client log lines for InferenceMax. Variant and benchmark_params accept bench_max_failed_requests (default 0 remains strict for CI). * feat(inference): add typed InferenceMax config loader (Phase 1) Move InferenceMax loading onto substitute_config and a typed InferenceMaxVariantConfig with legacy adapters for InferenceMaxJob until the driver is ported. * feat(inference): migrate InferenceMax configs to schema_version 1 (Phase 2) Flatten MI300X and MI355X variant configs to paths/model/container/roles/params/sweep and client.* threshold specs with enforce_thresholds false until recalibrated. * test(inference): wire inferencemax_single to typed config and sweep (Phase 2) Use variant_config and legacy adapter fixtures, parametrization from sweep.runs, and unit tests for load_variant and threshold adapters. * docs(inference): update InferenceMax config reference for schema_version 1 (Phase 2) Point loader and threshold docs at inferencemax_config_loader.load_variant and the client.* sweep cell format. * docs: fix stale dtni.config_loader references (Phase 1 tail) Point run-cvs-tests and dtni-dev-guide at cvs.lib.utils and inference/utils loaders. * feat(inference): rewrite InferenceMaxJob like VllmJob (Phase 3) Standalone driver uses Python-built vllm serve, vllm bench serve, and artifact parsing. Drop legacy InferenceBaseJob path and factory construction. * feat(inference): move InferenceMax server flags to roles.server.serve_args (Phase 3) MI300X and MI355X variants drop host-script and bench_serving params in favor of Python serve args. * test(inference): align inferencemax_single suite with VllmJob pattern (Phase 3) Add model_fetch, test_metric, and new InferenceMaxJob lifecycle. Update conftest and unit tests for typed config. * docs(inference): update InferenceMax reference for Phase 3 driver (Phase 3) Document Python serve, client.* metrics, and expanded lifecycle test stages. * chore(inference): remove unused inferencemax_host_scripts (Phase 5) Host script staging was dropped when InferenceMaxJob moved to Python-built vllm serve. * docs: clarify vllm_benchmark_scripts are legacy-only (Phase 5) InferenceMax and vllm_single build vllm serve in Python; this package remains for InferenceBaseJob paths. * docs(inference): rewrite InferenceMax reference for schema_version 1 (Phase 5) Replace legacy config/benchmark_params table with typed blocks and client.* thresholds. Document inferencemax_config_loader in AGENTS.md. * test(inference): add InferenceMaxJob parse_results unit test (Phase 5) Verify stock results artifact maps to client.* metrics via FakeOrch. * refactor(inference): rename inferencemax_single to inferencex_atom_single Adopt InferenceX ATOM as the framework identity while the suite is still internal. Renames the driver, config loader, pytest suite, variant configs, and documentation to inferencex_atom_single. * docs(plan): add InferenceX ATOM automation plan (MI300X + MI355X) Align the implementation plan with DTNI Validation Tracker workloads W1-W18, MI300X calibration seeds, and gsm8k accuracy gates. MI300X and MI355X variant dirs ship in parallel; MI355X thresholds stay record-only until lab calibration. Milestone 1 targets ATOM backend plus W1 on both arches. * docs(plan): add MI355X W1 calibration seeds from ATOM CI Document section 4.3 thresholds from ROCm/ATOM run 27912164002 and align M1 scope for dual-arch W1 calibration. * feat(inference): Phase 0 ATOM driver and W1 DeepSeek R1 variants Swap InferenceXAtomJob to atom.entrypoints.openai_server and atom.benchmarks.benchmark_serving when params.driver=atom. Add MI300X/MI355X W1 config+threshold dirs and cluster examples for deepseek-ai/DeepSeek-R1-0528. * feat(inference): complete Phase 0 W1 DeepSeek R1 recipe pins and MTP3 variants Add ix_recipes.json registry, ix_recipe_id/run_card in config loader, MTP3 variant dirs for MI300X/MI355X, copy-config dtni root, and run-card logging in tests. * feat(inference): add MI300X W1 DeepSeek R1 smoke variant Single-cell smoke config (C=128, 128 prompts) for shorter first lab validation before full atom_perf calibration. * FIX: underscore-prefixed keys (including _comment) are now stripped from the config dict before Pydantic validation, same as for thresholds. * Move InferenceX ATOM W1 configs to config_file layout and calibrate MI300X perf gates. Relocate DeepSeek R1 variants from input/dtni to the standard inference config tree, enable enforce_thresholds with lab-calibrated thresholds, and clear stale results.json before each benchmark run. * docs: replace section symbol with plain Section references. Use readable Section/Sections wording in plans and InferenceX ATOM variant config comments instead of the section sign character. * feat(inference): calibrate MI355X W1 thresholds from ATOM CI seeds. Apply the same 10% margin as MI300X lab gates to perf and MTP3 variant thresholds from ROCm/ATOM run 27912164002, document copy-config flow in README, and add config-loader unit tests. * fix(inference): address PR #229 review on inferencex_atom_single suite. Drop the post-launch sleep, fail model fetch on du errors instead of treating them as an empty cache, scope inf_res_dict to module like vllm_single, and document the bounded client poll timeout. * fix(inference): unbreak W1 Phase A threshold checks for ATOM artifacts. Pin DeepSeek W1 container names and derive failed/success_rate when ATOM omits failed; skip threshold enforcement for metrics the benchmark did not emit. * docs(plan): MI355X lab pending without blocking MI300X spine. Add Section 1.2 hardware policy, revise M1/Phase A exit criteria, and update milestone diagrams so MI355X confirmation is optional until nodes are available. * docs(plan): refresh IX-atom plan with accuracy, metrics, and CVS backlog. Align branch state and phases with ATOM driver reality, add accuracy test catalog, metric tiers, and platform enhancements; update W1 README lab notes. * docs(plan): add Section 12 coverage for variants, parity frameworks, and metrics. Document perf variant modes, workload-specific accuracy tests, inferencex_atom_vllm/sglang parity suites, supplemental and MTP metrics, and CI compare keys. * align: restore config_loader threshold discovery after dev/dtni rebase Re-apply merge resolutions from PR #233 alignment (dual threshold layout, vllm_single imports) that were lost when replaying commits onto e47df5a. * refactor(inference): deprecate InferenceMax legacy factory paths Route inferencemax_repo and framework=inferencemax to inferencex_atom with a warning. InferenceMax host jobs remain a placeholder that points callers at inferencex_atom_single. * refactor(inference): extract shared threshold sweep validation Add validate_thresholds_cover_sweep() for reuse by vllm_single and inferencex_atom loaders. Optional gated_metrics parameter allows framework-specific SLO sets. * refactor(inferencex): flatten configs and adopt vllm_single naming Rename W1 and GPT-OSS variant stems to {gpu}_inferencex-atom-single_{model}_{precision}[_{mode}] and remove per-variant subfolders. Update README and docs with new copy-config paths and W1 threshold gates for per-GPU throughput and tail latencies. * feat(inferencex): gate W1 per-GPU and tail latency metrics on ATOM path Add inferencex_atom_parsing with IX-specific GATED_METRICS (per_gpu_throughput, output_tput_per_gpu) without changing vllm_single. Wire InferenceXAtomJob and the suite to atom parsing; default metric_percentiles to 95,99 for p95 TPOT and p99 TTFT. * docs(inferencex): document ATOM-specific parsing vs vllm_single Clarify that W1 GATED_METRICS and output_tput_per_gpu live in inferencex_atom_parsing so vllm_single stays untouched until parity. * docs(inferencex): simplify variant README for lab users Drop vllm_single comparisons and internal threshold tables; keep naming pattern, variant list, and copy/run commands. * chore(utils): silence cluster placeholder resolution logs * test(config): align config_loader tests with sibling threshold discovery * feat(inferencex): add W1 metric tiers for tiered threshold gates * feat(inferencex): reuse server across sweep cells and config-driven waits * feat(inferencex): replace per-metric tests with tiered test_cell_metrics * chore(inferencex): enable server reuse on perf configs and shorter smoke waits * fix(inferencex): align cluster container names with variant configs * fix(inferencex): tighten W1 perf health gates when enforcing thresholds * test(inferencex): add GATED_METRICS parity and health gate coverage tests * test(inferencex): add GATED_METRICS parity and health gate coverage tests * docs(inferencex): update plan and README for flat layout and tiered gates * docs(inferencex): update plan and README for flat layout and tiered gates * chore(inferencex): trim expand_sweep docstring * docs(inferencex): clarify lab layout, launcher host, and results paths Document per-variant ~/input subdirs to avoid ambiguous threshold discovery, remote launcher vs GPU node prerequisites, and ~/cvs_results output paths. * docs(plan): prioritize multi-node as M5 after framework parity Elevate scaling to P1 milestone M5 immediately after M4 parity when hardware and suite recipes support nnodes>1; defer MTP+P2 widen to M6. * fix(inferencex): align W1 tpot tier with ATOM bench output Gate p99_tpot_ms instead of absent p95_tpot_ms, skip missing tier metrics in actuals, and recalibrate MI300X perf thresholds from the 2026-06-25 lab run. * chore(inferencex): use portable W1 perf thresholds on MI300X Replace per-node calibrated gates with conservative throughput floors and loose latency caps so healthy runs pass across lab nodes without recalibration. * test(inferencex): cover parse_results errors and client log failure paths * fix(inferencex): detect ATOM server early failures during wait_ready * refactor(inferencex): extract sweep reuse helpers and safer collection defaults * feat(inferencex): add explicit threshold_json paths to variant configs * chore(inferencex): polish conftest docs and simplify CLIENT_METRICS build * refactor(inferencex): inline atom_args and remove ix_recipe indirection * docs(plan): sync IX atom plan with inline atom_args config layout
…ation doc" This reverts commit 0b4f6fd.
* Add IX Run Deck reporting stack on top of dev/dtni. Rebased onto dev/dtni after IX-atom merge (#238). Keeps only Run Deck wiring and add-only doc updates; no unrelated IX-atom or utils doc churn. * Expand sweep charts with per ISL/OSL grouping and add vLLM preset. Group concurrency charts by sequence shape so multi-shape sweeps stay readable, add tail-latency chart presets, and wire vllm_single auto-registration. * fix(report): register inference suite preset before session bind Re-sync suite stem from pytest args at session start so auto-discovery works when tests run from the installed package. Log when no preset is registered instead of failing silently, and wire inferencex_atom explicitly in suite conftest as a fallback. * fix(report): link run-card pytest HTML and log via sibling basenames. * feat(report): add chart_comparison payload for multi-shape sweeps. Expose cross-shape series on a shared concurrency axis so the interactive viewer can render grouped comparisons without duplicating chart_series logic. * feat(report): add cross-shape sweep chart render helpers. Add grouped-bar and line comparison renderers plus shared tooltip CSS for static HTML chart surfaces. * feat(report): improve static run deck bar charts with axis grid. Add y/x gridlines, aligned tick labels, hover tooltips, and a viewer banner so cross-shape comparison stays in the interactive sidecar. * feat(report): expand interactive viewer with cross-shape comparison. Add grouped bar and line comparison modes, filter-aware heatmap and gates, and overview layout updates backed by chart_comparison payload tests. * feat(report): expose sweep_ttft_metric in viewer payload. Lets the interactive viewer default the throughput-vs-latency Y-axis to the suite TTFT metric. * feat(report): add percentile fans, gate margins, and heatmap toggle to viewer. Add P90/P95/P99 fan charts and gate margin vs concurrency, move throughput-vs-latency to the bottom, and let the heatmap switch between throughput and latency metrics. * refactor(report): drop unused static cross-shape chart renderers. Keep sweep_charts as tooltip CSS only; cross-shape comparison lives in the Chart.js viewer. * refactor(report): dedupe viewer charts and simplify multi-shape banner. Use sweep_has_multi_shape_comparison for the static deck banner, limit per-shape viewer lines to throughput and means, and document viewer chart surfaces in README. * Restore CI one-page summary HTML for inference suite reports. Bring back {report_basename}_summary.html generation and pytest bundle links, and extend the IX atom sample generator with a multi-shape variant. * Restore viewer baseline comparison and consolidate report docs. Bring back prev-run delta columns and payload wiring, remove ADDING_A_SUITE.md, and point suite/report docs at cvs/lib/report/README.md. * refactor(report): consolidate shared sweep shape and comparison helpers. Deduplicate shape grouping, prev-run comparison, preset builders, and viewer chart utilities without changing report behavior. * docs(inference): restore ADDING_A_SUITE.md with Run Deck step. Keep the existing suite guide on dev/dtni and retain the optional Step 8 report wiring section instead of deleting the file. * refactor(report): trim PR scope by deferring non-IX-atom extras. Remove the local sample generator and vLLM preset, shorten README, and drop unused prev-run HTML renderer and panel_shell helper. * refactor(report): rely on auto-register for IX-atom preset wiring. Drop explicit pytest_configure registration from the inferencex_atom suite conftest and expose a single stem-matched preset export. * refactor(report): dedupe cell-card CSS and drop chart_comparison pre-bake. Unify pytest and static cell-card themes, build cross-shape comparison only in the viewer, and trim overlapping unit tests. * fix(report): use bundle-relative links and omit config run-card notes. Write run deck artifacts into the pytest bundle folder, link pytest HTML and run logs correctly after zip extract, and stop auto-rendering variant run_card.notes in static HTML. * fix(report): address PR #244 review feedback. Merge duplicate pytest_configure hooks, escape HTML in report logs, and tighten report payload binding and tests. * fix(conftest): rename report bind fixture to match module scope. Addresses PR #244 fixture-scope review: name no longer implies session-scoped pytest fixture. * fix(report): align PR #244 inline review fixes. Use tryfirst pytest_configure, session-scoped report store with module bind merge, and key-is-None results column offset. * fix(report): sanitize absolute paths in link_or_text_html. Only treat ../ paths as bundle-relative so local filesystem paths reduce to basename in HTML reports.
…255) * refactor(inference): move shared suite helpers under inference/utils Colocate lifecycle, cache probe, and results table modules with other inference utilities and update import paths across IX atom suites and report wiring. * style: apply ruff formatting for make test fmt-check Run ruff format on 26 files across inference, report, and test modules so fmt-check passes before merging into dev/dtni. * fix(lint): resolve ruff check issues for make lint gate Move cache_probe import to module top, remove unused variables/imports, and add noqa for pytest side-effect imports so ruff check and pylint both pass. * refactor(inferencex_atom): colocate lib modules under inferencex_atom package * refactor(inferencex_atom): update imports for inferencex_atom package paths * moved lib import
…#241) * Updating sglanf for multinode Signed-off-by: amd-droy <droy@amd.com> * feat(gpu): add GPU polling loop, metrics, and threshold gates - gpu.py: parse_gpu_metrics, capture_gpu_metrics, _mean, agg_readings, poll_gpu_metrics - VllmJob.is_client_done(): non-raising completion predicate - vllm_single test: poll GPU while client runs, write gpu_poll.log, derive 5 metrics - _shared.py: Peak VRAM / Compute % / BW % columns in results table - test_gpu.py: TestMean, TestAggReadings, TestPollGpuMetrics unit test classes - threshold JSON: gpu.* placeholder SLO entries for all 5 cells - test_vllm_orch_parse: update threshold path + exclude gpu.* from client key guard * fix(vllm_single): add missing gpu_metrics_snap module-scope fixture The fixture was referenced in test_vllm_inference's parameter list but never defined, causing a setup Error before any inference ran. * fix(gpu): use exec_on_head for amd-smi; mkdir gpu_poll.log parent dir amd-smi is a host-side tool — running it via orch.exec() sends it into the container where it doesn't exist. Switch capture_gpu_metrics to orch.exec_on_head() so the command runs on the bare-metal node. Also ensure the out_dir exists before poll_gpu_metrics attempts to write gpu_poll.log, since the directory is created lazily by the job setup. Update unit test mocks from exec to exec_on_head to match. * fix(vllm_single): write gpu_poll.log to tmp then copy to node via exec_on_head out_dir is an NFS path on the node, not mounted on the devbox. Write the log to a local tempdir, then base64-encode it and push it to the node via exec_on_head so it lands in the bundle. * fix(gpu): move deferred imports to module level; fix test_gpu_metric rank Move import time/logging/pathlib from inside poll_gpu_metrics body to module top-level. Add test_gpu_metric at rank 4 in conftest sort table so it runs before test_teardown, not after. * docs(gpu): add gpu.py section to AGENTS.md and integration guide Add gpu.py API reference to cvs/lib/utils/AGENTS.md: public symbols, poll_gpu_metrics parameter table, 5-metric derivation table, required conftest fixtures (gpu_metrics_snap), two wiring patterns (sync poll / threaded poll), pytest_generate_tests parametrize branch, collection sort rank table, and gotchas (threshold key prefix, capture can raise, or-None semantics, full actuals for evaluate_all, GATED_METRICS). Add cvs/lib/utils/docs/gpu-metrics.md: user-facing integration guide covering the 5 derived metrics, polling lifecycle, 5-step integration walkthrough, gpu_poll.log format, failure/None handling table, and cross-references to ADDING_A_SUITE.md and threshold-kinds.md. * fix(vllm_single): write gpu_poll.log to local report dir so it lands in zip bundle Previously the log was written to a tempfile then uploaded to the NFS out_dir; because the zip plugin only bundles the local html report directory, the log never appeared in the run archive. Now it is written directly into the _test_html_dir folder (e.g. vllm_single_html/) so every run archive contains the poll log alongside the per-test HTML files. The NFS upload is kept for cluster-side access. Update gpu-metrics.md integration guide to match the correct log_path pattern and to describe where the log lands. * fix(vllm): guard is_client_done cat against missing log file orch.exec captures stderr; 'cat client.log' when the file does not yet exist emits 'No such file or directory' to stderr, which matches CLIENT_LAUNCH_FAIL_RE and causes is_client_done to return True immediately after run_client(). poll_gpu_metrics then exits with 0 readings. Replace bare cat with 'test -f ... && cat ... || true' so a missing file produces empty output (False) instead of a false launch-failure signal. * feat(gpu): add multi-node polling via nodes= param capture_gpu_metrics and poll_gpu_metrics now accept an optional nodes=[(label, phdl)] list. When provided, amd-smi is fanned out to all phdl handles and their GPU entries are merged before aggregation (sum VRAM, mean utilisation) — same output dict shape as the single-node path. Log lines are tagged with node labels; the summary block adds per-label used_vram lines. nodes=None (default) preserves the existing exec_on_head single-node path unchanged. 75 unit tests pass (66 existing + 9 new). * feat(sglang-disagg): wire GPU polling into performance benchmark test Threads poll_gpu_metrics alongside benchserv_test_random (Pattern B) using nodes=[('prefill-0', p_phdl), ('decode-0', d_phdl)] so both node groups are polled. is_done_fn cats the benchmark results log on the b_phdl node. gpu_poll_disagg.log lands in the local HTML report dir for bundle inclusion. Adds test_gpu_metric parametrized over the 5 GPU_METRICS keys, with gpu_metrics_snap fixture and pytest_generate_tests in conftest. test_gpu_metric ordered at rank 14 (after test_disagg_gpu_topology). * refactor(gpu): drop vllm_single wiring, keep gpu.py scope-focused The vllm_single suite has been substantially rewritten upstream since this branch split off; wiring GPU polling into it here was scope creep on a feature that's really about the gpu.py library + sglang-disagg integration. Revert all vllm_single/vllm test/lib/threshold changes to their current dev/dtni state and generalize the remaining vllm-specific references in gpu.py, its tests, and docs. * refactor(gpu): remove sglang-disagg wiring, gpu.py stays suite-agnostic Per feedback: this branch should not touch any suite. Revert the sglang-disagg GPU polling integration back to dev/dtni; only the gpu.py library, its unit tests, and docs remain. * refactor(gpu): move gpu.py to inference/utils, use orch.exec(hosts=) for multi-node Multi-node polling now targets host lists through the existing Orchestrator.exec(cmd, hosts=...) contract instead of requiring callers to hand-construct per-role Pssh/phdl handles, mirroring but generalizing the pattern in sglang_disagg_lib.py. Also fixes two poll_gpu_metrics bugs: is_done_fn() exceptions were previously misattributed as amd-smi failures, and multi-node polls ran two exec rounds (merged + per-node) instead of one. * refactor(gpu): move gpu.py back to cvs/lib/utils (suite-agnostic, not inference-only) gpu.py has no inference-specific logic — it shells out to amd-smi via an Orchestrator and parses/aggregates results. Training suites will need GPU polling too, so it belongs in cvs/lib/utils/ alongside the other framework-agnostic machinery (config_loader.py, verdict.py), not nested under cvs/lib/inference/. * addressing nits: failure cap bypass and incorrect docstrings --------- Signed-off-by: amd-droy <droy@amd.com> Co-authored-by: amd-droy <droy@amd.com>
* Integrating nodesmoke tier 1 tests using Primus cli * Fixed the braces * Updated the README * Added the robustness improvement Signed-off-by: Urvashi Tiwari <urtiwari.com>
…rametrized suite (#257) * Merge pull request #225 from ROCm/hnimrama/inferencemax-uplift Hnimrama/inferencemax uplift Refactors InferenceMax for the DTNI pytest layout (inferencemax_single): ContainerOrchestrator-based conftest, suite/threshold JSON loading, benchmark model selection, and tighter server/client lifecycle handling against current InferenceX upstream. Benchmarking: stop cloning third-party bench_serving; resolve benchmark_serving.py from the installed vllm package (BENCH_SCRIPT) for InferenceMax and vLLM single paths. Host-mounted server entrypoints live under cvs.lib.dtni.vllm_benchmark_scripts (vllm_serve_mi300x.sh); samples and docs use <changeme> container placeholders, legacy benchmark_script_repo called out as ignored, and volume_dict guidance avoids duplicate Docker :/workspace mounts. vLLM single (vllm_orch): align with dev/dtni completion and client-failure detection while keeping python3 "$BENCH_SCRIPT" invocation. Misc: optional run_plugin --log-file; sglang_disagg total_generated_tokens key; log redaction and small review fixes from PR feedback. Test with cvs run inferencemax_single (cluster + suite JSON, HF token) and spot-check vllm_single if configs touch shared modules. * Revert "Merge pull request #225 from ROCm/hnimrama/inferencemax-uplift" (#228) This reverts commit 5798f4e. * Hnimrama/ix atom (#238) AIMVT-244/Add ATOM framework and inference Deepseek r1 model * Restore InferenceMax uplift reverted by #228 Reverts commit 4a8425f, restoring the changes from PR #225 on dev/dtni. * fix(inference): run vLLM bench client with vLLM interpreter Probe python3.13..python3 for import vllm; export BENCH_PY and BENCH_SCRIPT. Use shlex.quote for docker exec bash -c. Align InferenceMax client completion with Serving Benchmark Result or End-to-end Latency. * fix(dtni): broaden vLLM benchmark script discovery Search site-packages and ancestor paths, verify the file is readable, and document vllm[bench] when wheels omit benchmarks/. * fix(inference): harden InferenceMax server startup and GPU mem env Use CVS_GPU_MEMORY_UTIL in sample config and serve script to avoid vLLM unknown-env warnings. Extend default readiness poll budget to 60 and grep full server logs so Uvicorn ready is not missed after long model loads. * fix(dtni): fall back to vllm bench serve when benchmark script is absent Wheels often omit vllm/benchmarks; resolve the driver via eval exports, run python -m vllm.entrypoints.cli.main bench serve when needed, and fail fast on missing-script log patterns in InferenceMax and base polling. * fix(dtni): clamp bench random-range to max_model_length vLLM random workloads scale (ISL+OSL)*(1+r); clamp ratio when it would exceed MML, pass --temperature 0 for greedy parity, and forward --metric-percentiles in InferenceMax and vllm_single clients. * fix(inference): extend InferenceMax bench client poll budget Read client_poll_count and client_poll_wait_time from benchmark_params (defaults 50/60), document them and fix the inferencemax.rst table, and surface the keys in sample MI300X/MI355X configs. * feat(inference): add bench_max_failed_requests cap and completion-first polling Gate benchmark success on Failed requests only after the summary is present; tail more client log lines for InferenceMax. Variant and benchmark_params accept bench_max_failed_requests (default 0 remains strict for CI). * feat(inference): add typed InferenceMax config loader (Phase 1) Move InferenceMax loading onto substitute_config and a typed InferenceMaxVariantConfig with legacy adapters for InferenceMaxJob until the driver is ported. * feat(inference): migrate InferenceMax configs to schema_version 1 (Phase 2) Flatten MI300X and MI355X variant configs to paths/model/container/roles/params/sweep and client.* threshold specs with enforce_thresholds false until recalibrated. * test(inference): wire inferencemax_single to typed config and sweep (Phase 2) Use variant_config and legacy adapter fixtures, parametrization from sweep.runs, and unit tests for load_variant and threshold adapters. * docs(inference): update InferenceMax config reference for schema_version 1 (Phase 2) Point loader and threshold docs at inferencemax_config_loader.load_variant and the client.* sweep cell format. * docs: fix stale dtni.config_loader references (Phase 1 tail) Point run-cvs-tests and dtni-dev-guide at cvs.lib.utils and inference/utils loaders. * feat(inference): rewrite InferenceMaxJob like VllmJob (Phase 3) Standalone driver uses Python-built vllm serve, vllm bench serve, and artifact parsing. Drop legacy InferenceBaseJob path and factory construction. * feat(inference): move InferenceMax server flags to roles.server.serve_args (Phase 3) MI300X and MI355X variants drop host-script and bench_serving params in favor of Python serve args. * test(inference): align inferencemax_single suite with VllmJob pattern (Phase 3) Add model_fetch, test_metric, and new InferenceMaxJob lifecycle. Update conftest and unit tests for typed config. * docs(inference): update InferenceMax reference for Phase 3 driver (Phase 3) Document Python serve, client.* metrics, and expanded lifecycle test stages. * chore(inference): remove unused inferencemax_host_scripts (Phase 5) Host script staging was dropped when InferenceMaxJob moved to Python-built vllm serve. * docs: clarify vllm_benchmark_scripts are legacy-only (Phase 5) InferenceMax and vllm_single build vllm serve in Python; this package remains for InferenceBaseJob paths. * docs(inference): rewrite InferenceMax reference for schema_version 1 (Phase 5) Replace legacy config/benchmark_params table with typed blocks and client.* thresholds. Document inferencemax_config_loader in AGENTS.md. * test(inference): add InferenceMaxJob parse_results unit test (Phase 5) Verify stock results artifact maps to client.* metrics via FakeOrch. * refactor(inference): rename inferencemax_single to inferencex_atom_single Adopt InferenceX ATOM as the framework identity while the suite is still internal. Renames the driver, config loader, pytest suite, variant configs, and documentation to inferencex_atom_single. * docs(plan): add InferenceX ATOM automation plan (MI300X + MI355X) Align the implementation plan with DTNI Validation Tracker workloads W1-W18, MI300X calibration seeds, and gsm8k accuracy gates. MI300X and MI355X variant dirs ship in parallel; MI355X thresholds stay record-only until lab calibration. Milestone 1 targets ATOM backend plus W1 on both arches. * docs(plan): add MI355X W1 calibration seeds from ATOM CI Document section 4.3 thresholds from ROCm/ATOM run 27912164002 and align M1 scope for dual-arch W1 calibration. * feat(inference): Phase 0 ATOM driver and W1 DeepSeek R1 variants Swap InferenceXAtomJob to atom.entrypoints.openai_server and atom.benchmarks.benchmark_serving when params.driver=atom. Add MI300X/MI355X W1 config+threshold dirs and cluster examples for deepseek-ai/DeepSeek-R1-0528. * feat(inference): complete Phase 0 W1 DeepSeek R1 recipe pins and MTP3 variants Add ix_recipes.json registry, ix_recipe_id/run_card in config loader, MTP3 variant dirs for MI300X/MI355X, copy-config dtni root, and run-card logging in tests. * feat(inference): add MI300X W1 DeepSeek R1 smoke variant Single-cell smoke config (C=128, 128 prompts) for shorter first lab validation before full atom_perf calibration. * FIX: underscore-prefixed keys (including _comment) are now stripped from the config dict before Pydantic validation, same as for thresholds. * Move InferenceX ATOM W1 configs to config_file layout and calibrate MI300X perf gates. Relocate DeepSeek R1 variants from input/dtni to the standard inference config tree, enable enforce_thresholds with lab-calibrated thresholds, and clear stale results.json before each benchmark run. * docs: replace section symbol with plain Section references. Use readable Section/Sections wording in plans and InferenceX ATOM variant config comments instead of the section sign character. * feat(inference): calibrate MI355X W1 thresholds from ATOM CI seeds. Apply the same 10% margin as MI300X lab gates to perf and MTP3 variant thresholds from ROCm/ATOM run 27912164002, document copy-config flow in README, and add config-loader unit tests. * fix(inference): address PR #229 review on inferencex_atom_single suite. Drop the post-launch sleep, fail model fetch on du errors instead of treating them as an empty cache, scope inf_res_dict to module like vllm_single, and document the bounded client poll timeout. * fix(inference): unbreak W1 Phase A threshold checks for ATOM artifacts. Pin DeepSeek W1 container names and derive failed/success_rate when ATOM omits failed; skip threshold enforcement for metrics the benchmark did not emit. * docs(plan): MI355X lab pending without blocking MI300X spine. Add Section 1.2 hardware policy, revise M1/Phase A exit criteria, and update milestone diagrams so MI355X confirmation is optional until nodes are available. * docs(plan): refresh IX-atom plan with accuracy, metrics, and CVS backlog. Align branch state and phases with ATOM driver reality, add accuracy test catalog, metric tiers, and platform enhancements; update W1 README lab notes. * docs(plan): add Section 12 coverage for variants, parity frameworks, and metrics. Document perf variant modes, workload-specific accuracy tests, inferencex_atom_vllm/sglang parity suites, supplemental and MTP metrics, and CI compare keys. * align: restore config_loader threshold discovery after dev/dtni rebase Re-apply merge resolutions from PR #233 alignment (dual threshold layout, vllm_single imports) that were lost when replaying commits onto e47df5a. * refactor(inference): deprecate InferenceMax legacy factory paths Route inferencemax_repo and framework=inferencemax to inferencex_atom with a warning. InferenceMax host jobs remain a placeholder that points callers at inferencex_atom_single. * refactor(inference): extract shared threshold sweep validation Add validate_thresholds_cover_sweep() for reuse by vllm_single and inferencex_atom loaders. Optional gated_metrics parameter allows framework-specific SLO sets. * refactor(inferencex): flatten configs and adopt vllm_single naming Rename W1 and GPT-OSS variant stems to {gpu}_inferencex-atom-single_{model}_{precision}[_{mode}] and remove per-variant subfolders. Update README and docs with new copy-config paths and W1 threshold gates for per-GPU throughput and tail latencies. * feat(inferencex): gate W1 per-GPU and tail latency metrics on ATOM path Add inferencex_atom_parsing with IX-specific GATED_METRICS (per_gpu_throughput, output_tput_per_gpu) without changing vllm_single. Wire InferenceXAtomJob and the suite to atom parsing; default metric_percentiles to 95,99 for p95 TPOT and p99 TTFT. * docs(inferencex): document ATOM-specific parsing vs vllm_single Clarify that W1 GATED_METRICS and output_tput_per_gpu live in inferencex_atom_parsing so vllm_single stays untouched until parity. * docs(inferencex): simplify variant README for lab users Drop vllm_single comparisons and internal threshold tables; keep naming pattern, variant list, and copy/run commands. * chore(utils): silence cluster placeholder resolution logs * test(config): align config_loader tests with sibling threshold discovery * feat(inferencex): add W1 metric tiers for tiered threshold gates * feat(inferencex): reuse server across sweep cells and config-driven waits * feat(inferencex): replace per-metric tests with tiered test_cell_metrics * chore(inferencex): enable server reuse on perf configs and shorter smoke waits * fix(inferencex): align cluster container names with variant configs * fix(inferencex): tighten W1 perf health gates when enforcing thresholds * test(inferencex): add GATED_METRICS parity and health gate coverage tests * test(inferencex): add GATED_METRICS parity and health gate coverage tests * docs(inferencex): update plan and README for flat layout and tiered gates * docs(inferencex): update plan and README for flat layout and tiered gates * chore(inferencex): trim expand_sweep docstring * docs(inferencex): clarify lab layout, launcher host, and results paths Document per-variant ~/input subdirs to avoid ambiguous threshold discovery, remote launcher vs GPU node prerequisites, and ~/cvs_results output paths. * docs(plan): prioritize multi-node as M5 after framework parity Elevate scaling to P1 milestone M5 immediately after M4 parity when hardware and suite recipes support nnodes>1; defer MTP+P2 widen to M6. * fix(inferencex): align W1 tpot tier with ATOM bench output Gate p99_tpot_ms instead of absent p95_tpot_ms, skip missing tier metrics in actuals, and recalibrate MI300X perf thresholds from the 2026-06-25 lab run. * chore(inferencex): use portable W1 perf thresholds on MI300X Replace per-node calibrated gates with conservative throughput floors and loose latency caps so healthy runs pass across lab nodes without recalibration. * test(inferencex): cover parse_results errors and client log failure paths * fix(inferencex): detect ATOM server early failures during wait_ready * refactor(inferencex): extract sweep reuse helpers and safer collection defaults * feat(inferencex): add explicit threshold_json paths to variant configs * chore(inferencex): polish conftest docs and simplify CLIENT_METRICS build * refactor(inferencex): inline atom_args and remove ix_recipe indirection * docs(plan): sync IX atom plan with inline atom_args config layout * feat(dtni): add vllm_distributed CVS suite for 2-node MI300X multinode inference Introduces vllm_distributed, a new CVS inference validation framework for 2-node MI300X clusters running vLLM with tensor parallelism (TP=8) and pipeline parallelism (PP=2) across 16 GPUs total via the multiprocessing distributed executor backend. New files: cvs/lib/inference/vllm_distributed.py VllmDistributedJob class: - build_server_cmd applies 5 in-container patches per run to fix upstream vLLM bugs in the rocm/ufb-private nightlies image: Patch 0: delete stale multiproc_executor.pyc and core.pyc Patch 0b: replace assert in multiproc_executor.py:collective_rpc (rpc_broadcast_mq is None on PP follower nodes); return safe default instead of crashing Patch 1: guard _initialize_kv_caches() for follower nodes; use dummy KVCacheConfig(num_blocks=1) to skip collective_rpc Patch 2: stub Scheduler() with _F on follower nodes to skip KVCacheManager/HybridKVCacheCoordinator assert Patch 3: fix get_supported_tasks() to return ("generate",) for follower nodes (SupportedTask is Literal, not Enum) - is_ready() / wait_ready(): per-poll readiness with fatal-log detection - run_client(): bench serve head-only via exec_on_head - postcheck(): validates server log, client log, result file - collect_logs(): zips node logs and HTML artifacts cvs/lib/inference/utils/vllm_distributed_config_loader.py config schema cvs/lib/inference/unittests/test_vllm_distributed.py 52 unit tests cvs/tests/inference/vllm_distributed/ pytest suite cvs/input/config_file/inference/vllm_distributed/ config + thresholds Modified files: cvs/core/orchestrators/container.py openssh-server fallback install for images without sshd; per-cmd timeout cvs/lib/inference_lib.py register vllm_distributed framework cvs/lib/inference/unittests/test_vllm_orch_parse.py fix threshold JSON path Validated on 10.245.135.15 (g21u43, head) + 10.245.135.115 (h16u07, worker) with amd/Llama-3.1-70B-Instruct-FP8-KV, ISL=1000 OSL=1000 concurrency=16. Signed-off-by: Atul Nair <Atul.Nair@amd.com> * style: apply ruff formatting to vllm_distributed suite files Signed-off-by: Atul Nair <Atul.Nair@amd.com> * fix(dtni): address review feedback on vllm_distributed suite - Revert cvs/core/orchestrators/container.py: the openssh-server fallback install should not be in core; the ufb-private image already ships sshd (confirmed by v7a7 validation pass) - Replace VllmDistributedJob alias with direct use: test suite imported VllmDistributedJob as VllmJob; now uses the class name directly - Scrub personal references from config: threshold_json absolute path, master_addr IP, and GLOO/TP/NCCL_SOCKET_IFNAME NIC name replaced with <changeme> placeholders - Remove VllmDistributedJob from InferenceJobFactory registry: VllmDistributedJob's constructor (orch, variant, ...) is incompatible with create_job's calling convention (c_phdl, s_phdl, ...) so the entry was unreachable dead code * fix(dtni): remove test_setup_sshd from vllm_distributed suite vLLM with --distributed-executor-backend mp uses PyTorch distributed (TCPStore on master_addr:master_port) for inter-node rendezvous -- no SSH between containers is required. The sshd test was cargo-culted from MPI-based suites and fails on images that do not ship openssh-server. * feat(vllm): unified vllm suite replacing vllm_single + vllm_distributed New files: - cvs/lib/utils/ib_discovery.py: discover_ib_hca_names() via ibv_devinfo -l; fails loudly on empty nodes or asymmetric HCA lists across nodes - cvs/lib/inference/utils/vllm_config_loader.py: unified VariantConfig for single-node (nnodes=1, pp=1) and distributed (nnodes>1, pp>1); ib_netdev required when nnodes>1; cell_key emits PP= segment only when pp>1 - cvs/lib/inference/vllm_job.py: unified VllmJob; distributed flags added iff nnodes>1; run_client/wait_client_complete/parse_results use exec_on_head; IB devices (ib_hcas, ib_netdev) written into env script; no runtime patches - cvs/tests/inference/vllm/vllm.py: unified suite with test_discover_topology lifecycle step; single-node skips discovery; distributed validates config ib_hca_devices list at preflight - cvs/input/config_file/inference/vllm/: single and distributed config templates Modified: - cvs/tests/inference/vllm/conftest.py: switch to vllm_config_loader, add test_discover_topology to rank map, fix hf_token for remote=0 * fix(vllm): address suite bugs found during core42 validation run Three issues surfaced by live 2-node run on 10.245.135.11/13: 1. ib_discovery: add /sys/class/infiniband sysfs fallback when ibv_devinfo is absent from the image. ROCm vLLM images ship without libibverbs-dev but sysfs always reflects the same HCA names NCCL_IB_HCA needs. 2. vllm_job: fix is_ready() and wait_ready() to use per-rank log paths. Previously broadcast self.server_log (node0 path) to all nodes — node1 always got exit_code!=0 (file not found), causing a 60-minute timeout on any follower failure. Now checks _rank_log(rank) on each host and fails fast via _check_early_failure() each poll iteration. 3. vllm: skip test_setup_sshd entirely. vLLM distributed uses --distributed-executor-backend mp + NCCL over host network; no inter-container sshd is needed (unlike MPI-based suites). The vLLM image also does not ship openssh-server. * fix(vllm): headless worker ranks + head-only readiness for distributed Worker ranks (rank > 0) launch with --headless, and is_ready() only greps the head rank for the startup pattern when nnodes > 1 (worker ranks have no API server and never log 'Application startup complete'). * perf(vllm): reuse server across cells with identical server args The unified vLLM suite restarted vllm serve (full weight reload + warmup) for every sweep cell, even when consecutive cells differed only in concurrency — a client-only knob that never changes the server command. For a multi-concurrency sweep this paid one ~9-minute weight reload per cell for no reason. - Add VllmJob.server_signature(): the rank-agnostic server identity (argv minus --node-rank, plus the env map), excluding client-only knobs. Cells with the same signature can share one running server. - test_vllm_inference reuses the live server when the signature matches the one recorded on the lifecycle object, skipping stop/start/wait_ready; a failed cell clears the recorded signature so the next cell does a clean bringup. - Fix a duplicate --max-model-len: _server_argv emitted a derived value AND the config's serve_args value, so vllm saw the flag twice (config silently won). Now the derived value is only emitted when serve_args does not pin it. Adds test_vllm_job_server_reuse.py covering the dedup and signature behavior (concurrency-invariant; ISL/OSL-sensitive when max-model-len is derived). * chore(vllm): remove legacy vllm_single + vllm_distributed suites The unified vllm suite (cvs/tests/inference/vllm/vllm.py) parametrizes both single-node (nnodes=1) and multinode distributed (nnodes>1, PP across nodes) runs, fully replacing the two legacy suites. Nothing outside the deleted files imported the legacy classes/loaders. Removed: - suites: tests/inference/vllm/vllm_single.py, tests/inference/vllm_distributed/ - lib: vllm_single.py, vllm_distributed.py - loaders: inferencing_config_loader.py, vllm_distributed_config_loader.py (both superseded by vllm_config_loader.py) - legacy unit tests: test_vllm_orch_parse.py, test_vllm_distributed.py, test_inferencing_config_loader.py - sample configs under input/config_file/inference/vllm_{single,distributed}/ Kept _shared.py (the unified suite imports it) and fixed two now-stale docstrings. `cvs list` shows only `vllm`; remaining unit tests pass. * fix(vllm): create cell out_dir in run_client for the server-reuse path The server-reuse path skips build_server_cmd (which did the per-cell `mkdir -p out_dir`), so a reused cell's client wrote client.log/results into a directory that never existed -> 'No such file or directory' and the cell failed. run_client now ensures its own out_dir on the head node, so it is correct whether the server was freshly built or reused. Regression test added. * fix(vllm): pass --trust-remote-code to bench client when server enables it The bench client loads the tokenizer from --model to count tokens. Models whose tokenizer_config declares a custom tokenizer via auto_map (e.g. Kimi-K2.6) fail this load with ValueError unless trust-remote-code is set. The server already honored serve_args trust-remote-code; mirror it on the client so the same tokenizer loads. Validated on 2-node Kimi-K2.6-MXFP4 (TP8xPP2): 3 cells, 0 failed requests, 297/530/830 tok/s. * style(vllm): satisfy ruff lint + format gate Remove 4 unused imports and apply ruff formatter to the report and inference modules introduced on this branch, so `make build` (fmt-check + lint) passes. No behavior change. Verified: ruff check clean, ruff format clean, pylint 10/10, 415 unit tests + 38 CLI tests pass. * feat(vllm): surface server log content and validate serve_args at load time - _check_early_failure: add emit_tail param; precheck and warmup calls log the tail -30 snapshot at INFO so startup/engine-load lines appear in the CVS capture without re-emitting on every poll iteration - wait_ready: log readiness poll iter=N/M at each iteration for progress visibility during the up-to-60-minute poll window - _flatten_serve_args: False values now omit the flag entirely (previously emitted --flag False which argparse rejected as unrecognized argument) - EARLY_FAILURE_RE: extended with argparse error patterns so a server CLI parse failure raises on the precheck call instead of spinning to the cap - RoleServer: add field_validator for serve_args.log-level; invalid values raise ValidationError at config-load time rather than at server launch - Tests: FakeOrchWithOutput + _make_job_for_check helper; 7 new cases covering False omission, True flag-only, log-level pass-through, emit_tail logging, CLI parse error raise, and the log-level validator * fix(vllm): restore inferencing_config_loader.py deleted in error The rebase onto dev/dtni's PR #244 dropped this module believing it was legacy vllm_single-only, but dev/dtni's InferenceX ATOM suite (added independently via PR #244) imports Sweep/SeqCombo/GoodputSlo/Run from it via inferencex_atom_config_loader.py. Restoring it and its test file verbatim from dev/dtni fixes the ModuleNotFoundError. * feat(vllm): add Ray distributed-executor-backend support Adds ray as a distributed-executor-backend option for multi-node vLLM serving (nnodes>1, pp=1), alongside the existing mp backend. Bootstraps a Ray cluster (head + workers) before vllm serve, runs vllm serve only on the head, and tears down via ray stop on stop_server. * fix(inference): remove stale duplicate left by dev/dtni rebase cvs/lib/inference/inference_suite_lifecycle.py was renamed to cvs/lib/inference/utils/inference_suite_lifecycle.py upstream (dev/dtni #255). Rebasing onto dev/dtni with -X ours left a corrupted duplicate at the old path with no remaining references; delete it. * fix(vllm): stop overriding configured num_prompts with magic constants test_vllm_inference always overrode variant_config.params.num_prompts (a documented config field, default 3200) with a hardcoded concurrency*20/50 heuristic keyed on osl >= 8192. Use the configured value directly instead. * fix(inference): remove old-path duplicates reintroduced from a stale rebase dev/dtni PR #255 moved cache_probe.py, inferencex_atom_orch.py, inference_suite_results_table.py, inferencex_atom_config_loader.py, and inferencex_atom_parsing.py to their inferencex_atom/ and utils/ locations. This branch's rebase onto dev/dtni left the old-path copies in place alongside the moved ones, landing divergent duplicate source modules. No code references the old paths; deleting them. * fix(vllm): use correct exec() output key in fatal-error check detailed=True exec() results carry the log text under "output", not "stdout" (every other call site in this file already uses "output"). The wrong key meant FATAL_LOG_RE could never match, so a fatal RuntimeError in the server log would go undetected until the full readiness poll cap instead of failing fast. * style(vllm): apply ruff format to PR files Wraps two over-width lines in test_vllm_job_ray_backend.py to satisfy ruff format --check; no logic changes. * style(utils): apply ruff format/lint fixes to gpu.py + test_gpu.py Pre-existing drift unrelated to the vllm unification PR: ruff format line-wrapping and E401 multiple-imports-on-one-line in test_gpu.py. No logic changes. --------- Signed-off-by: Atul Nair <Atul.Nair@amd.com> Co-authored-by: Hamna Nimra <hnimrama@amd.com>
* feat(vllm): wire unified vLLM suite into inference report engine The generic inference report engine (cvs/lib/report/) already landed on this branch via dev/dtni, but no suite preset existed for `vllm`, so `cvs run vllm` never rendered a run deck. Add the missing preset plus its vLLM-specific tier/column glue: - vllm_parsing.py: VLLM_RESULTS_COLUMNS (mirrors _shared.py's console table), METRIC_TIERS/METRIC_TIER_ORDER seeded from the existing GATED_METRICS set so the report's gate matrix partition exactly matches what the suite already enforces, and tier_metric_specs(). - report/presets/vllm.py: registers VLLM_REPORT_CONFIG under stem "vllm" (cvs/tests/inference/vllm/vllm.py), with explicit lifecycle labels overriding the builder defaults -- this suite records topology_discovery instead of sshd_setup/client_complete. - tests/inference/vllm/conftest.py: remove the suite-local lifecycle table hookwrapper, now redundant with root conftest's attach_inference_suite_lifecycle_table once the preset is registered (confirmed byte-for-byte equivalent rendering logic). - unittests/test_vllm_report_preset.py: pins the tier partition bijection with GATED_METRICS and the auto-register wiring. Verified with an offline dry render (synthetic VariantConfig + inf_res_dict via write_report()): HTML/JSON/viewer artifacts generate, gate matrix tiers compute correctly, and the lifecycle timeline shows only the stages this suite actually records. * fix(vllm): correct stale hook reference in lifecycle docstring The docstring pointed to pytest_runtest_makereport, which no longer renders these rows after the vLLM suite was wired into the generic inference report engine; attach_inference_suite_lifecycle_table does it now.
…ting (#258) * fix(vllm): account for pipeline-parallel size in per_gpu_throughput * fix(vllm): account for pipeline-parallel size in per_gpu_throughput * fix(inferencex_atom): pass pp=1 to to_client_metrics after pp became required ATOM has no pipeline-parallel concept (single-node TP only), so pp=1 preserves the prior per_gpu_throughput numeric behavior. Also widen the Ray-backend regression test to assert pp explicitly across pp=1/2 so a broken pp passthrough can't slip through silently again. * Revert "fix(inferencex_atom): pass pp=1 to to_client_metrics after pp became required" This reverts commit a2eb3c4. * feat(vllm): wire GPU metrics polling into the vLLM inference suite Adds test_gpu_metric (one HTML row per GPU metric per sweep cell), GPU pre/post-load VRAM snapshots and background amd-smi polling during test_vllm_inference, and extends threshold-coverage validation to gate gpu.* metrics alongside client.* metrics. amd-smi runs fine from inside the benchmark container (GPU device files are passed through), so gpu.py calls orch.exec/exec_on_head directly like every other command in the suite -- no host-vs-container bypass routing needed. * chore(vllm): remove drive-plan ledger artifact accidentally committed to PR spec-a1-ledger.md is an internal drive-plan tracking file, not part of the shipped test suite; it doesn't belong under cvs/lib/inference/unittests/. Signed-off-by: Atul Nair <Atul.Nair@amd.com> * fix(vllm): default pp="1" in to_client_metrics to unbreak ATOM Making pp a required keyword-only arg broke InferenceX ATOM at runtime: its wrapper (inferencex_atom_parsing.to_client_metrics) calls this function without pp, so every ATOM run raised TypeError in parse_results(). ATOM has no pipeline-parallel concept, so defaulting pp to "1" here reproduces the pre-fix tp-only formula for those callers while vLLM call sites keep passing pp explicitly. Addresses blocking review feedback on PR #258. Signed-off-by: Atul Nair <Atul.Nair@amd.com> * chore(vllm): remove unused gpu_metrics_snap fixture Never consumed by any test -- test_vllm_inference tracks GPU snapshots via local pre_snap/post_snap variables instead. Leftover from the generic gpu-metrics.md fixture template that the vLLM implementation diverged from. Addresses non-blocking review feedback on PR #258. Signed-off-by: Atul Nair <Atul.Nair@amd.com> * fix(vllm): replace thread-based GPU poller with detached remote script poll_gpu_metrics() spawned a second OS thread that shared the orchestrator's gevent-based SSH transport with the main thread's client-log-tail polling, causing a real HW-observed SessionError(OutOfBoundaryError()) race. Replace it with start_gpu_poller()/stop_and_collect_gpu_poller(): a detached remote background script per node writes amd-smi snapshots to a file, read back via ordinary sequential exec/exec_on_head calls, extending to real multi-node round-aligned collection. * style(vllm): ruff-format the new gpu poller test classes * fix(gpu): scope poller /tmp files by user and clean them up after collection start_gpu_poller()/stop_and_collect_gpu_poller() write a script and log file to /tmp on each polled node, keyed only by a sanitized run_id (e.g. a pytest node id). On shared hardware nodes (SSH-based orch.exec, not a per-container /tmp), two users running a similarly-named test can collide on the same path, and the files were never removed afterward -- the same failure mode reported for Fremont's conda-based CVS runs where /tmp is mounted through from the host. Scope the marker by the local SSH user (getpass.getuser()) in addition to run_id, truncate any stale log at launch instead of only appending, and rm -f both the script and log on each node after stop_and_collect_gpu_poller reads them back -- following the same never-raises degrade pattern already used for the pkill broadcast and read-back. --------- Signed-off-by: Atul Nair <Atul.Nair@amd.com>
Adds VllmJob.probe_openai_endpoints(), reusing the shared OpenAIProbe helper already used by the sglang suite, but driven through orch.exec_on_head instead of docker exec/Pssh. Wires it in as a new lifecycle stage (test_openai_compatible_smoke) that brings up a short-lived server at a small fixed cell and checks GET/POST /v1/models, /v1/chat/completions, /v1/completions, and structured JSON output before the full sweep runs.
test_openai_compatible_smoke derived max-model-len from the unrelated _SMOKE_ISL/_SMOKE_OSL sweep-cell constants (296 tokens), but the OpenAI-compatible probe sends its own fixed-content requests -- the structured-output-book probe alone needs ~50 prompt + 256 response tokens, exceeding that budget and failing every run with HTTP 400. Set serve_args["max-model-len"] explicitly so the flawed derivation is bypassed for this test path.
_check_early_failure() only tails the server log during startup, so a mid-benchmark crash (e.g. EngineDeadError) currently vanishes from the captured pytest output. Add VllmJob.dump_server_log(), mirroring the existing dump_client_log() pattern but per-rank, skipping ray-backend headless workers (rank > 0) which never run vllm serve. Wire it into test_vllm_inference's failure path only, tracking lifecycle.live_server_job so the dump targets whichever job actually owns the running server -- on the reuse path (cells that only differ by concurrency) that's an earlier cell's job, not the current one.
* Orchectration and Reporting. Signed-off-by: amd-droy <droy@amd.com>
…te integration (#268) * feat(inference): add AccuracyTask/AccuracyConfig schema for accuracy harness First unit of the lm-eval-harness accuracy evaluation system: the config.json-side task selection schema, split from threshold/gating values which will be joined in at runtime by a later unit. * feat(inference): add lm_eval_parsing.py auto-discover projector Second unit of the accuracy harness: pure JSON -> {scalar: float} projector for lm-eval-harness results.json payloads. Walks every numeric metric with no per-task registry, so group tasks (e.g. RULER's per-seq-length metrics) fall out of the same walk. * feat(inference): add lm_eval_job.py command builder + orchestration runner * feat(inference): add test_accuracy_eval lifecycle stage (step 4) * Wire AccuracyConfig and test_accuracy_eval into vllm and inferencex_atom suites Adds the accuracy field to both suites' VariantConfig classes (defaulting to an empty AccuracyConfig so existing configs load unchanged), imports the shared test_accuracy_eval lifecycle stage into each suite's test module, and places it in collection order right after the perf-metric stage. * fix(accuracy): address adversarial review findings for accuracy harness - exclude the top-level "accuracy" threshold key from sweep-cell coverage checks (was tripping the extra-key/typo detector); delegate vllm_config_loader's inline check to the shared validator - gate accuracy threshold evaluation on enforce_thresholds, matching test_metric's existing record-only convention - route chat-template tasks to /v1/chat/completions instead of always using /v1/completions - wrap exec_on_head tuple-unpack and JSON parse failures in run_accuracy_tasks as RuntimeError instead of letting raw ValueError/JSONDecodeError propagate - select the newest results*.json by mtime instead of an arbitrary find ordering, guarding against stale results from a prior run * fix(accuracy): pass --apply_chat_template to lm_eval for chat-template tasks build_lm_eval_cmd switched the model backend/endpoint to local-chat-completions for tasks with apply_chat_template=True but never passed lm-eval's own --apply_chat_template flag, so lm-eval sent plain-string prompts and the chat-completions client asserted on every request. Verified on a live 2N DeepSeek-R1 accuracy run where this aborted mmlu_pro and blocked all subsequent tasks in the same test_accuracy_eval invocation. * revert(accuracy): un-wire test_accuracy_eval from inferencex_atom_single Scope this feature to vllm only until it's been validated against ATOM hardware. Removes the accuracy field from InferenceXAtomVariantConfig, the test_accuracy_eval import/collection-order entry in the ATOM suite, and the corresponding ATOM-specific unit tests. The shared lm_eval_job.py/lm_eval_parsing.py/accuracy_config.py machinery and test_accuracy_eval itself are untouched -- vllm's wiring is unaffected. * feat(accuracy): parametrize test_accuracy_eval by task Each accuracy task now gets its own pytest node (test_accuracy_eval[<id>]) instead of one collapsed row covering every configured task, matching the test_metric/test_gpu_metric per-metric row convention. Task nodes are gated independently: a run failure or threshold violation in one task no longer sets the shared lifecycle.failed flag, so sibling tasks still execute rather than being skipped by a prior task's outcome. pytest_generate_tests parametrizes accuracy_task from config.json's accuracy.tasks, including the empty-list case (auto-skips a single node, same UX as before). * test(accuracy): use gsm8k instead of mmlu as the default task fixture mmlu spans 57 subjects and is unnecessarily slow for a placeholder task in schema-validation tests; gsm8k exercises the same construction paths without implying a real eval choice. * fix(accuracy): propagate server env, trust_remote_code, and exit-code checks to lm-eval lm_eval was launched without sourcing /tmp/server_env_script.sh, so HF_HUB_CACHE/HF_TOKEN set during server setup never reached it -- tokenizer resolution could fail on a fresh head node even though it happened to work on hosts with a pre-warmed cache. Source the script the same way VllmJob's client-launch path already does. Models with custom tokenizer code (Qwen, ChatGLM, Phi, MPT, ...) need trust_remote_code=True or they fail to load; add it unconditionally to model_args since it's a no-op for models that don't need it. run_accuracy_tasks treated any results*.json under the output dir as success, even a stale one from a prior run, without checking lm-eval's own exit status. ContainerOrchestrator.exec_on_head/DockerRuntime.exec_on_head didn't support detailed=True (unlike their sibling exec() methods and BaremetalOrchestrator.exec_on_head), so exit codes were unreachable on the container-runtime path the vLLM/accuracy suite actually uses -- add the missing detailed param end to end and check exit_code before falling through to the results-file lookup.
…efill p50/p95) (#274) * feat(vllm): add Prometheus /metrics-derived latency metrics (queue/prefill p50/p95) Scrapes vLLM's own /metrics endpoint before/after each client run, diffs the queue-time and prefill-time histograms to isolate one cell's observations, and interpolates p50/p95 quantiles matching PromQL's histogram_quantile(). Gated as prom.* metrics via a parallel GATED_PROM_METRICS set (mirroring GATED_GPU_METRICS) to keep out of the locked client.* tiering partition. * fix(vllm): include gpu/prom metric rows in report cell-card extras test_gpu_metric and test_prom_metric rows were missing the pytest-html cell-card attachment that test_metric rows get, since row_card_test_names only listed test_metric. * fix(vllm): pin test_prom_metric in the lifecycle order rank test_prom_metric was missing from pytest_collection_modifyitems' rank dict, so it fell through to the default rank (99) and collected after test_teardown -- running the prom metric assertions against an already torn-down container instead of alongside test_metric/test_gpu_metric. * fix(vllm): clamp histogram_quantile to the highest finite bound in +Inf bucket PromQL cannot linearly interpolate past the last finite bucket boundary, so it clamps a quantile that falls in the unbounded "+Inf" bucket to that boundary instead of extrapolating to infinity. Our implementation returned +Inf in that case, diverging from the PromQL parity the docstring claims. Under the overload scenarios these metrics target, a p95 exceeding every finite bucket would surface as `inf` ms instead of a finite value (and risk `Infinity` in strict-JSON serialization if gated). The one exception (a "+Inf"-only histogram with no finite boundary to clamp to) still returns +Inf, matching PromQL. * fix(vllm): stop requiring every gated client/gpu/prom metric in threshold.json _check_thresholds_cover_sweep previously raised (under enforce_thresholds: true) if a cell's threshold entry omitted ANY client.*/gpu.*/prom.* gated metric, making the full 20+-metric union mandatory per cell. Downstream evaluation (test_metric/test_gpu_metric/test_prom_metric) already treats an absent spec as "don't gate this metric" -- the completeness check existed only at load time and served no evaluation-time purpose. This meant a user who only wants to gate a couple of metrics (e.g. just prom.queue_time_p50_ms) was forced to also author specs for every other client/gpu/prom metric, or drop to enforce_thresholds: false and lose gating entirely. Adding prom.* to the gated union (this PR) would have made this worse for any downstream threshold.json with enforce_thresholds: true, since it would now also need 4 new prom.* specs per cell just to keep loading. Drops the completeness check; keeps the still-needed sweep-cell-coverage check (every sweep cell must have SOME threshold entry, cell keys must not be stray/typo'd). * style(vllm): ruff format prom-metrics files (line-length wraps) * docs(vllm): remove dangling VLLM_PROMETHEUS_METRICS_SPEC.md references The spec doc was never added to the repo/PR, so its ~10 citations across comments/docstrings pointed readers at a nonexistent file. Drop the references, keep the underlying rationale inline.
* Added optional Tier 2 node_smoke support to preflight via config (tier2_perf: true). When enabled, preflight passes --tier2-perf and thresholds for: GEMM TFLOPS floor HBM D2D bandwidth local multi-GPU RCCL all-reduce Updated: node_smoke.py, preflight config/schema/docs, report summary, and unit tests. Signed-off-by: Urvashi Tiwari <urtiwari.com> * Fixed the default setting Signed-off-by: Urvashi Tiwari <urtiwari.com> --------- Signed-off-by: Urvashi Tiwari <urtiwari.com> Co-authored-by: Urvashi Tiwari <urtiwari.com>
…284) leaderboard_math_hard imports math_verify when building its task prompt templates. The install guard requested only the `api` extra, so the module was absent and the task died with ModuleNotFoundError after the server was already up -- observed on GLM-5.2-FP8 (TP8/PP2, ROCm 7.14), where 8 of 10 accuracy tasks scored and math_hard failed on the missing dependency. Request `lm-eval[api,math]`, which pulls math-verify, sympy>=1.12 and the pinned antlr4 runtime. Also replace the `pip list | grep lm_eval` presence check with an import probe. The old guard tested only that some lm_eval existed, not that it carried the needed extras: on an image preinstalling bare lm-eval it would short-circuit, skip the math extra, and reproduce this same failure while the install line looked correct. Probing `import lm_eval, math_verify` fails closed instead. Unit-tested only -- the in-container install path is not exercised by the suite and still needs a hardware rerun of leaderboard_math_hard to confirm.
…#278) * Stop double-logging every remote command's output pssh's own host_logger emits every remote stdout/stderr line, tagged with the host. Upstream keeps it quiet behind a NullHandler unless enable_host_logger() is called -- which CVS never does -- but cvs/lib/globals.py binds `log` to the ROOT logger, so propagation delivers those lines to CVS's handlers anyway. Pssh._process_output then logs each line a second time. Measured on a 2-node DeepSeek R1 TP16/PP1 vLLM run: 3,747,478 duplicate pairs, 49.8% of all lines in a 491 MB cvs.log. Detaching the third-party logger keeps the _process_output copy, which is the one that honors print_console and can therefore be suppressed for bulk-data commands. Host attribution is preserved by the existing "Host == <host> ==" banner that _process_output prints per host. * Thread print_console through the orchestrator and runtime layers Pssh.exec has always accepted print_console, but ContainerOrchestrator and DockerRuntime dropped it, so a caller asking for a quiet bulk read still got every line logged. Passing the kwarg from a call site raised TypeError before this change. Added as a keyword argument defaulting to True in last position, mirroring the existing `detailed` parameter, so every current call site is unaffected. Also aligned the ContainerRuntime protocol, BaremetalOrchestrator, and the enroot stub. MultiProcessPssh (what cvs/core actually gets from the parallel_ssh_lib shim) already accepts and forwards print_console on both its sharded and delegating paths, so no change was needed below the runtime layer. Tests pin forwarding on all three DockerRuntime exec paths and all three BaremetalOrchestrator paths -- a dropped kwarg fails silently, so the default-stays-verbose case is pinned too. * Stop logging raw amd-smi JSON from the GPU metrics reads The GPU poller writes amd-smi JSON to a file on each node every 15s; stop_and_collect_gpu_poller then cats the whole accumulated file back and parses it. The cat output was logged in full. On a 2h14m 2-node DeepSeek R1 TP16/PP1 run that was 7,457,416 lines / 486 MB from two cat calls -- 99.1% of a 491 MB cvs.log. The pre/post snapshot calls added another 2.1 MB. Nothing is lost. The digest side-file (gpu_poll_isl*_osl*_conc*.log, 23 KB) already carries every value any consumer reads, as do the peak_gpu_memory_mb / gpu_compute_util_pct / gpu_bandwidth_util_pct aggregates. The suppression is applied inside gpu.py, where "this is bulk data" is locally true, rather than left to each caller. * Emit each vLLM log once instead of two or three times _check_early_failure tails each rank's server log on every readiness poll and, under emit_tail, re-logs it with host+rank labels. With the transport also logging it, each line landed up to three times. Across 75 polls that was 30,159 emitted lines carrying only 974 distinct ones -- 97% repeats. The same pattern appeared in dump_client_log, dump_server_log, parse_results, and the client-completion poll: each reads content and then either emits it under its own label or parses it. Suppressing transport-level logging on these reads keeps exactly one copy, and it is the best-labeled one -- dump_server_log's crash dump (PR #270) is now easier to read, not harder. dump_client_log runs on every exit path of the completion poll, so the per-iteration tail added nothing. * Preserve failure diagnostics the quieting silenced Suppressing console echo on the vLLM log reads removed the only copy of some output that mattered on failure paths: - test_openai_compatible_smoke never dumped its server log, relying on the poll loop's echoed tails. A bringup timeout now left nothing from the failing server behind; it dumps like the sweep does. - parse_results raised on a malformed artifact without quoting it, so the content was no longer anywhere in the log. Also gives ContainerOrchestrator.exec_on_head the detailed parameter its baremetal counterpart already had. build_mpi_cmd calls exec_on_head with detailed=True, which raised TypeError on the container path; unreachable today since distribute_using_mpi has no in-tree caller, but this PR is already reshaping that signature. The host-subset forwarding test asserted against the same MagicMock for both the subset and all-hosts handles, so it passed even with the branch it guards removed. It now pins the constructed host list. * Filter pssh host_logger instead of clearing its propagate flag The previous fix set propagate=False on pssh.host_logger. That works in a plain process but not under pytest, which is how CVS actually runs: _pytest.logging.catching_logs attaches its capture handler to root AND to every non-propagating logger, so clearing propagate makes pytest attach directly to host_logger and the duplicate line survives. A filter drops the record before any handler is consulted, so it holds however the handler was attached. It also survives a level reset, which a setLevel(CRITICAL) approach would not. The old unit test asserted on the propagate flag in a plain process, so it verified the mechanism intended rather than the outcome wanted. The new tests assert no pssh.host_logger record is captured -- once with a handler bolted straight onto the logger, once inside a real catching_logs -- plus a guard that other loggers are still captured. * Sync the Orchestrator ABC and pin container-layer forwarding Review follow-up on #278. The ABC advertised a narrower contract than every class implementing it: `print_console` (this PR) and `detailed` (pre-existing) were accepted by BaremetalOrchestrator and ContainerOrchestrator but declared by neither abstract method. Python's abstractmethod enforces method names only, never parameter lists, so nothing raised and no test caught it -- the gap only surfaces when someone writes a new backend from the ABC and omits the kwarg, and a dropped print_console is silent: the command still works, it just logs hundreds of MB again. test_abc_signature_matches_implementation compares the ABC's parameters against the concrete implementation's so this drift cannot recur quietly. The container forwarding pins mirror the existing baremetal ones onto the path the vLLM suite actually runs, and which previously dropped both kwargs. They read arguments by keyword or position, so they keep holding if the runtime signature grows; verified by mutation (removing either forward fails four of them with a named-argument message, not IndexError). * Address review nits; back out the ABC signature change Revert the Orchestrator ABC signature widening from 95907bd. The ABC's sole subclass is BaremetalOrchestrator, which ContainerOrchestrator inherits from, so the concrete signatures are already the contract every call site sees. The ABC/impl drift also pre-dates this PR. Three review nits: - globals.py: name the host_logger suppression filter instead of an anonymous lambda, so it is identifiable in logging.getLogger('pssh.host_logger').filters when debugging log routing on a live node. - vllm_job.py: repr() the artifact snippet carried into the parse_results error. The artifact can be a stack trace or an HTML error page; raw newlines there break up CI output and pasted ticket bodies. - container.py: forward detailed/print_console to the runtime by keyword rather than positionally. Also mirror the exec/exec_on_head forwarding pins from test_baremetal.py into test_container.py -- container is the path the vLLM suite actually runs on, and the one that previously dropped both kwargs. Validated by mutation: deleting the forwarding args in container.py fails 4 of the 6 pins.
all commits squashed. Hopefully clean merge.
BaremetalOrchestrator.exec, BaremetalOrchestrator.setup_env and DockerRuntime.exec build a throwaway Pssh when the caller passes a host subset and drop it on return. When the wrapped exec times out, the per-host greenlet in ParallelSSHClient.cmds never finishes; its callable is a bound method of the client, so the client stays reachable, SSHClient.__del__ never runs and the sshd session outlives the object. vLLM readiness polling issues subset execs per rank up to 60 times a run, so sessions accumulate on the shared account until sshd's per-user limit rejects unrelated logins. destroy_clients() was 'del self.client', a no-op against this. It now kills pending command greenlets and calls _disconnect() on each host client, tolerating per-host errors so one dead host cannot abort teardown of the rest. The three subset call sites tear their temporary handle down in try/finally, since the timeout path is the only one that leaks and cleanup cannot depend on a clean return. setup_env has no callers today, but it is an abstractmethod on the base orchestrator, so an implementation could reach it; it gets the same treatment for consistency. Long-lived orch.head/orch.all are reused rather than rebuilt and measure flat across repeated timeouts, so they are left untouched. Verified against a live sshd: 8 timing-out subset execs through BaremetalOrchestrator grew sessions 1..8 before and hold flat at 1 after. Clean execs never leaked.
Removes the pytest-html Value/Unit result columns for vllm, sglang, and inferencex_atom suites. vllm and inferencex_atom previously populated real per-metric values; sglang's columns were always blank since no sglang test sets metric_value/metric_unit. Hooks and the shared helper are commented out (not deleted) rather than removed.
* feat(inferencex_atom): add multinode params and scaling efficiency metric Extend the variant schema with nnodes, pipeline parallelism, rendezvous addresses, and optional single-node throughput baseline for scaling.efficiency_pct. Multinode sweep cells include PP and NNODES in the threshold cell key. * feat(inferencex_atom): run distributed ATOM serve across cluster ranks When params.nnodes is greater than one, launch one server per host with per-rank distributed executor flags, poll all ranks for failures, and run the benchmark client on the head node only. Single-node runs are unchanged. * feat(inferencex_atom): add MI300X 2-node cluster and W1 perf_multi variant Ship example cluster JSON, config, and seeded thresholds for the W1 DeepSeek R1 FP8 multinode reference cell. Thresholds stay record-only until lab confirmation. * test(inferencex_atom): cover multinode job routing and config loading Extend FakeOrch with exec_on_head and per-host exec tracking; add unit tests for distributed server launch, head-only client, scaling metric, and perf_multi variant schema. * docs(inferencex_atom): document multinode cluster and variant params Add lab run instructions for the 2-node W1 perf_multi stem and describe nnodes, pipeline parallelism, rendezvous, and scaling baseline fields. * feat(orchestrator): install openssh-server when image lacks sshd Multinode container runs need in-container sshd on port 2224; some ROCm images ship without openssh-server. Fall back to apt install before start. * fix(inference): restore test_launch_container in inference_suite_lifecycle The function body was left orphaned after the du_bytes import, causing an IndentationError when pytest collected inferencex_atom_single. * fix(inferencex_atom): use ATOM-native multinode serve instead of vLLM PP flags Replace unrecognized vLLM distributed CLI args with ATOM SPMD env and -dp when TP allows; TP8 two-node runs independent replicas per host. Harden multinode sshd setup (iproute2, pgrep fallback) and health-check all ranks. * fix(inferencex_atom): isolate ATOM multinode argv from vLLM distributed flags * config(inferencex_atom): expand multinode sweep to 5 shapes x conc 16/64/128 * feat(inferencex_atom): enforce multinode scaling thresholds in gates and report Enable threshold enforcement for the 2-node perf variant, gate scaling.efficiency_pct as a dedicated metric tier, and surface it in the IX Run Deck preset. * fix(inference): repair syntax error in test_launch_container Restore the newline between the docstring and timing call that was lost during rebase conflict resolution. * fix(inference): tolerate CollectReport in html_metric_table_row Collection failures emit CollectReport objects without user_properties; avoid AttributeError so pytest-html surfaces the real collection error. * refactor(inferencex_atom): rename suite from inferencex_atom_single Drop the misleading _single suffix so multinode runs use cvs run inferencex_atom; keep inferencex_atom_single as a deprecated config framework alias. * docs(inferencex_atom): run make install before activating venv Clarify lab setup order so Makefile can manage .cvs_venv without an active shell inside it. * style(inferencex_atom): fix ruff format and lint on multinode changes Apply ruff formatting and resolve lint issues in the 12 multinode-touched Python files so fmt-check and ruff check pass on this branch delta. * feat(inferencex_atom): add MI355X multinode cluster and perf_multi variant Add mi355x_atom_multi cluster file, perf_multi config/threshold scaffold, README runbook, and config-loader unit test mirroring the MI300X 2-node path. * config(inferencex_atom): add MI300X baseline perf sweep variant DeepSeek R1 FP8 sweep over 1K/1K and 8K/1K at concurrency 4 through 256 with portable gated thresholds. * config(inferencex_atom): add MI355X baseline perf sweep variant Same 1K/1K and 8K/1K concurrency matrix as MI300X with record-only threshold seeds until lab calibration. * test(inferencex_atom): cover baseline perf sweep config loading Assert 14-cell matrix, max_model_length, and threshold coverage for MI300X and MI355X baseline sweep variants. * docs(inferencex_atom): document baseline perf sweep runbook Add variant table entries and MI300X lab commands for the 14-cell 1K/1K and 8K/1K concurrency sweep. * config(inferencex_atom): point models_dir at /home/models Set HF hub cache to /home/models and bind-mount it into ATOM containers across all shipped variant configs. * config(cluster): note /home/models expectation for IX-atom templates Document the shared model cache path on GPU nodes in MI300X and MI355X atom cluster examples. * docs(inferencex_atom): document /home/models cache layout Update runbook, reference docs, and config loader test for the shared model path and container bind mount. * config(cluster): consolidate IX-atom cluster templates Replace four GPU/topology-specific cluster files with one inferencex_atom_cluster.json and document matching node_dict length to variant nnodes. * feat(inferencex_atom): add multinode baseline sweep and drop inferencex_atom_single alias * refactor(inferencex_atom): rename config stems from inferencex-atom-single to inferencex-atom Align shipped variant and threshold filenames with the inferencex_atom suite naming. * chore(inferencex_atom): refresh multinode configs, docs, and loader tests * fix(inferencex_atom): calibrate perf_multi C=16 thresholds from lab run 20260722 * fix(container): fail fast when openssh-server is missing from image Remove runtime apt-get install of openssh-server and raise a clear error when /usr/sbin/sshd is absent so multinode jobs fail at setup time. * feat(inferencex_atom): add vllm_atom and sglang drivers for true PP multinode Extend the orchestrator with framework-coordinator paths that preserve vLLM distributed executor flags and SGLang PP launch, while standalone atom keeps SPMD data parallel for scale-out without native pipeline parallel. * test(inferencex_atom): cover vllm_atom PP2 and sglang distributed launch Add loader and orchestrator unit tests for PP=2 cell keys, vLLM executor flags with headless workers, and SGLang dist-init launch wiring. * config(inferencex_atom): switch multinode variants to vllm_atom PP=2 Replace driver=atom multinode configs with vllm_atom pipeline parallel, serve_args, ib_netdev, and PP=2 threshold cell keys for W1 scaling runs. * config(inferencex_atom): add SGLang PP=2 multinode W1 variant Ship a SGLang pipeline-parallel multinode config and seed thresholds for MI300X W1 scaling parity alongside the vLLM-ATOM path. * docs(inferencex_atom): document vllm_atom and sglang PP multinode paths Clarify that true pipeline parallel requires a framework coordinator driver, not standalone atom, and update variant tables for PP=2 multinode configs. * docs(inferencex_atom): align plan and runbooks with PP=2 driver model Document vllm_atom and sglang as multinode pipeline coordinators, refresh the automation plan M5 status, and expand variant README lab prerequisites. * docs(cluster): improve inferencex_atom cluster template for multinode labs Clarify rank-0 GPU head IP vs jumphost and default cluster_id_ed25519 key path. * Rename inferencex_atom configs to flat single/distributed layout. Drop dedicated smoke configs so multinode and single-node variants follow the same vLLM-style naming convention. * Fix multinode vLLM-ATOM readiness and align configs with flat layout. Check only rank-0 startup on the head node for PP runs, extend distributed poll timeouts for DeepSeek cold start, and update threshold_json paths plus lab docs for the renamed single/distributed configs. * Fix multinode NCCL/Gloo networking for inferencex_atom. Wire IB HCA discovery and GLOO_SOCKET_IFNAME like the vLLM suite so PP=2 lab runs get correct NCCL env exports. * fix(container): repair setup_sshd docstring and drop sshd preflight check. Single-node runs already skip sshd setup; remove the redundant openssh-server gate that blocked minimal images unnecessarily. * Auto-discover socket netdev for multinode inferencex_atom. Resolve GLOO/NCCL socket interfaces from cluster IPs in test_discover_topology so labs no longer hand-enter ib_netdev each run. * Coerce legacy mlx5 ib_netdev configs to auto discovery. Strip orchestrator-managed NCCL/Gloo env keys so old lab configs load without blocking socket netdev discovery. * Resolve multinode fabric lazily when topology test is skipped in smoke -k runs. * Probe multinode fabric on host OS, not inside container. * Fix host topology probes: valid bash ip syntax and ibv banner parsing. * Recalibrate MI300X multinode thresholds from 20260723 lab run. Set ISL=512/OSL=512 CONC=16 gates to 80% of measured throughput and scaling efficiency, and 110% of measured TTFT/TPOT tails. * Recalibrate CONC=16 TTFT gates from 20260724 multinode run. Raise mean_ttft and p99_ttft maxes to 110% of measured values (402ms and 15441ms). * Raise W1 multinode max_model_length to 8192 for 2k sweep cells. Fixes vLLM Bad Request failures when random 2k/1k prompts exceed 4096 context. * Raise W1 multinode client poll cap for long 1k/2k sweep cells. * Recalibrate W1 multinode thresholds from 20260728 full sweep at 80% measured. * Rename inferencex_atom suite to atom across code, configs, and docs. Drops InferenceX naming so multinode vLLM-ATOM work stands alone without DeepSeek V4 Pro commits on this branch. * Fix atom suite collection after dev/dtni merge. Restore ATOM_RESULTS_COLUMNS export and guard pytest-html row hook against CollectReport objects that lack user_properties. * Remove deprecated inferencex_atom results column alias. * Fix atom report preset chart series for merged ReportChartSeries API. Pass scaling metrics through full_metric unchanged and drop the removed metric_key constructor arg. * Fix sshd port probe when ss is missing from container images. Fall back to netstat and bash /dev/tcp for multinode setup_sshd validation. * Fix sshd listen probe and report cell_build import after rename. PSSH detailed exec returns output not stdout, so setup_sshd falsely failed despite /dev/tcp OK. Restore cell_build verdict import to cvs.lib.utils.verdict. * Restore dev/dtni report wiring after rename cherry-pick. The atom rename commit pulled in rundeck and pytest_hooks imports that do not exist on dev/dtni, breaking conftest load. Restore report infrastructure from the pre-merge backup while keeping atom presets and fixes. * Fix multinode topology discovery for container atom smoke. Restore exec_on_host on ContainerOrchestrator, correct netdev shell command substitution (not arithmetic expansion), drop duplicate exec_cmd_list, and align atom report lifecycle labels. * Fix netdev discovery bash subshell syntax. Use IF=\ instead of invalid \ip ...; brace groups that bash rejects in command substitution. * Address PR review: restore inference_lib placeholders, sglang stop, docstring
Fixes fmt-check failures without changing runtime behavior.
Drop the outdated branch-scoped plan doc now that multinode atom work has landed on dev/dtni.
…scovery (#303) * test(atom): align multinode unit tests with session key and fabric discovery Update server_session_key fixtures with nnodes/pp/master fields and stub IB topology in build_server_cmd tests so FakeOrch runs match the lab lifecycle path. * test(atom): fix RUF012 ClassVar lint in recording orch helper
* docs(atom): add inference suite README for tests/inference/atom Document suite layout, lifecycle, sweeps, metric tiers, and quick-start commands in the same style as the JAX MaxText training suite README. * docs(atom): rewrite config README in jaxmaxtext reference style Restructure the input-config guide with file inventory, must-change table, config/threshold reference, and condensed lab workflow; point to the new tests/inference/atom README. * docs(atom): scope READMEs to ATOM drivers and trim clutter Drop sglang/vllm placeholder references, migration notes, and redundant related-doc links from the config and suite guides.
* reformatted to pass ruff formatter check in the CI * removed the unused line from the code Signed-off-by: Urvashi Tiwari <urtiwari.com> Co-authored-by: Urvashi Tiwari <urtiwari.com>
…ibuted) (#298) * feat(vllm): add MI325X workload config set (single + distributed) Adds 14 vllm inference workloads for MI325X, each as a single/distributed pair -- 28 configs with 28 sibling threshold files, plus a README. Topology follows the requested uniform shape: single is TP8/PP1/1 node, distributed is TP8/PP2/2 nodes. Several workloads in the source list specify TP=4; TP=8 is used throughout per the directive and the deviation is documented in the README. Every sweep carries the three concurrency-16 shapes (1k1k, 1k8k, 8k1k) as sequence_combinations, but only 1k1k is referenced by sweep.runs, so a run executes exactly one cell. random_range_ratio is 0.0 so ISL/OSL are exact. enforce_thresholds is false on all configs and each threshold file covers the selected cell with permissive placeholders for all 25 gated client.* metrics, satisfying the coverage check without warnings until MI325X is calibrated. Environment-specific values (model.id, container.image, models mount, ib_netdev, master_addr) are redacted to <changeme>. All 28 configs verified to load cleanly through vllm_config_loader.load_variant with zero warnings, emitting the expected cell keys: single ISL=1024,OSL=1024,TP=8,CONC=16 distributed ISL=1024,OSL=1024,TP=8,PP=2,CONC=16 * fix(vllm): use per-model TP for MI325X workloads DeepSeek V4 Flash, Kimi K2.6, Kimi K2.5 and gpt-oss-20b run at TP=4 per the source workload list, instead of the previous uniform TP=8. TP=8 is unchanged for the other ten workloads. Distributed variants keep PP=2 across 2 nodes regardless of TP, so a TP=4 distributed run uses 4 GPUs per node. Threshold cell keys follow automatically (ISL=1024,OSL=1024,TP=4,CONC=16 and the PP=2 form). All 28 configs still load through load_variant with zero warnings. * docs(vllm): require a routable IPv4 when picking ib_netdev An interface that merely exists is not enough: NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME and TP_SOCKET_IFNAME all take this name, and gloo fails engine init with "Unable to find address for: <name>" when the interface is DOWN or has no IPv4. Point at `ip -o -4 addr show` so the check catches that case. * fix(vllm): set fp8 kv-cache for DeepSeek V4 workloads DeepseekV4ForCausalLM uses the fp8_ds_mla attention layout, which asserts "only supports fp8 kv-cache, got auto" and fails engine initialization under the default kv-cache dtype. Hit on hardware with DeepSeek-V4-Flash-FP8; V4-Pro shares the architecture. * fix(vllm): enable AITER for DeepSeek V4 workloads DeepSeek V4's sparse attention indexer has no non-AITER ROCm path and refuses to initialize without VLLM_ROCM_USE_AITER=1. Confirmed on hardware: with it unset the worker dies with "Sparse attention indexer ROCm path is only supported on AITER". * fix(vllm): enable AITER + GPU_ARCHS for sparse-attention workloads GLM-5.1/5.2 (GlmMoeDsaForCausalLM) reach the same sparse attention indexer as DeepSeek V4 via deepseek_v2.py -> mla.py -> sparse_attn_indexer.forward_hip, which raises 'only supported on AITER' without VLLM_ROCM_USE_AITER=1. AITER's JIT kernel build then fails with "One of GPU archs of [''] is invalid" because GPU_ARCHS is empty in the image; MI325X is gfx942. Scoped to the 4 stems whose model config carries the DSA indexer keys (index_topk / index_head_dim / index_n_heads). * feat(vllm): add gpu/prom threshold placeholders, zero all values Threshold files covered only the client.* family. Add the gpu.* (5) and prom.* (4) families so every metric the suite records has a visible slot, taking 32 specs per file. Set every value to 0 across all three families. These are uncalibrated placeholders, not measurements. Note a 0 on a max/max_ms kind is an impossible bound, so enabling enforce_thresholds before calibration fails loudly rather than passing silently -- the previous 1e9 values did the opposite. Kinds follow the metric's unit: max for MB/s counts, min for utilization floors, max_ms only for genuine milliseconds. * feat(vllm): add accuracy placeholder blocks to MI325X workload set Accuracy is the one gated family split across both files: config.json's `accuracy.tasks` selects which lm-eval tasks run, and threshold.json's `accuracy` block holds the gating values keyed by task id, then by lm-eval metric key. Because those keys derive from the task ids chosen in the config, they cannot be pre-enumerated the way client.*/gpu.*/prom.* were -- so both blocks ship empty, as visible slots rather than absent ones, with the fill-in shape documented inline and in the README. No behavior change: `pytest_generate_tests` reads `raw.get("accuracy", {})`, so an empty block and an absent one both parametrize on an empty list and the node is auto-skipped. The `accuracy` threshold key is exempt from the sweep-cell coverage check via NON_SWEEP_THRESHOLD_KEYS. Verified: all 28 configs load through `load_variant` with 0 failures and 0 warnings; the exemption is discriminating, not vacuous (a typo'd "accuracie" still warns). 74 accuracy unit tests pass. Full suite 1084 tests / 4 failures, identical to baseline. fmt-check and lint are red on 23 pre-existing Python files, identical count with and without this change (proven by stash) -- this diff is JSON and Markdown only. * refactor(vllm): retarget workload config set from MI325X to MI300X Renames the directory, all 56 file names, and the gpu_arch / threshold_json / container.name fields. Content-neutral: normalizing the arch token shows zero files differing beyond the name, so no threshold, sweep or serve_arg changed. GPU_ARCHS stays gfx942 -- MI300X and MI325X are both CDNA3 and report the same arch, so the AITER env for the four sparse-attention workloads is correct for either, and the runs already validated on MI325X hardware remain applicable. Verified: all 28 configs load through load_variant with 0 failures and 0 warnings; every threshold_json resolves to a file that exists; leak scan clean and all 28 configs still carry <changeme>. Unit tests 1084 / 4 pre-existing failures, fmt-check 23 files, both identical to baseline. * chore: drop local gate marker from the workload config branch .last-gate-pass is a per-worktree marker the pre-push hook touches; it was swept in by a git add -A and is not deliverable content. * docs(vllm): clarify num_prompts, threshold coverage, and suite cross-links Review feedback on the MI300X workload set. num_prompts is 320 while the shipped vllm/ examples and the schema default are 3200, which reads as a full-length benchmark unless you check. Say so in the README and in each config's _comment, with the reason and how to undo it. The Thresholds section listed the 32-slot grid without saying whether the loader wants it. It only checks cell coverage -- an absent metric spec means "don't gate this metric" -- so the grid is calibration convenience, not a requirement. Add a See also pointing at the vLLM suite reference and how-to, so the threshold-kind and multinode material lives in one place. --------- Co-authored-by: Atul Nair <atnair@amd.com>
…training suites (#300) * [JAX] Adding Jax-maxtext training test refactor Updating the Jax maxtext with Orchestrator instead of direct pssh. * [JAX] Updated Time per Step - mean/p50/p95(ms) * [JAX] Added Scaling Efficiency calculation for training. * [JAX] Added Convergence check and validation loss/training loss check. * [JAX] Added Loss curve plot for the Training loss. It also checks for the loss curve slop for decrease in slope. Added jaxmaxtext unittests. * [JAX] Renamed config files and paths to match naming conventions. * [JAX] Added one metric results html file for all metrics tests. * [JAX] Enabled training sweeps run for few parameters. * [JAX] Separate test files for single & distributed tests. Also updated the sweep label (short) name with 3 params: [PRECISION+SEQLEN+BATCH] ex: BF16-SL4096-B3 * [JAX] Enabled error patterns from config file. List of error patterns to be scanned in the training log is added as user editable list via config file. If this list i empty, it will fallback to the existing list of error patterns in the code. * feat(JAX) Added README files for jax test and config files. * [JAX] Updated config files and ruff formatting fixes. * [JAX] Fixed self review comments 1. num_gpus no longer assumes 8 GPUs/node Added gpus_per_node: int = 8 to TrainingConfig (schema), and the job now computes self.num_gpus = self.num_nodes * self.gpus_per_node from config (via getattr, default 8) with a comment explaining it feeds tokens_per_sec_total → scaling efficiency. Made it explicit and editable in all three configs ("gpus_per_node": 8), and documented it in both READMEs (config table + a "must change per cluster" row). * [JAX] Review comment fix and updates 1. Signle and distributed test files are now proper pytest files with direct pytest methods with clear docstring instead of tricky bind method. 2. Version-flexible train script path (train_script → train_script_paths) 3. User-namespaced scratch dir (/tmp/jax → /tmp/<user>/jax) /tmp/{user-id}/jax/TRAINING_LOGS/ * [JAX] Address PR #300 review comments * [JAX] Add node dmesg error scan around training Gated on a new training.verify_dmesg config flag (default true; disable on clusters without passwordless sudo for dmesg). Best-effort: an infra failure of the scan itself is logged and swallowed so it never masks the training result. verify_lib is imported lazily so the training lib stays importable without the broader utils stack. Wired into _common.training_run after parse_results. Signed-off-by: Saravanan Solaiyappan <saravanan.solaiyappan@amd.com>
pull_image was the only one of twelve docker invocations in DockerRuntime
that hardcoded `sudo` instead of orchestrator.sudo_prefix(). Two
consequences:
- Credential mismatch. registry_login() does honor sudo_prefix(), so on
a cluster without passwordless sudo it authenticates as the SSH user
while the pull runs as root, which reads an empty
/root/.docker/config.json. A private image then fails with "pull
access denied" despite a successful login.
- Lost -n. Bare `sudo` drops the non-interactive flag, so a password
prompt blocks until the 600s timeout instead of failing fast.
This matters now that setup_containers() pulls unconditionally whenever
check_image_exists() reports the image missing, which puts the pull on
the startup path of every container-orchestrator suite, including the
RVS and AGFHC health suites.
Also fixes the two unit tests failing on this branch:
- test_pulls_image_when_missing_before_run asserted a bare
"docker run" prefix against a MagicMock orchestrator, whose
sudo_prefix() returned a Mock rather than a string. Stub it
explicitly and assert the rendered prefix.
- test_run_test_omits_log_file_when_not_set did not patch
_validate_json_config, so the new pre-flight check called
sys.exit on the mock config path. Patch it, matching the two
sibling tests.
Adds test_pull_uses_sudo_prefix_not_hardcoded_sudo to pin the fix for
both the sudo and no-sudo cluster shapes.
Signed-off-by: Atul Nair <Atul.Nair@amd.com>
* Document the unified vLLM suite The vLLM docs described a schema the suite no longer has: a `config` + `benchmark_params.<model>` layout, a `vllm_single` suite name, and a config path under inference/vllm_single/<variant>/. None of those exist on dev/dtni, and the reference page named a config file that ships nowhere in the repo. - Add reference/configuration-files/vllm.rst covering the current schema: every config block, the four distinct "backend" settings, the container and Docker keys, sweep cell keys and server reuse, all six threshold kinds and both coverage axes, and all four metric namespaces (37 client / 5 gpu / 4 prom / accuracy) - Add how-to/run-vllm-benchmarks.rst as the task-shaped entry point, including the mp-vs-ray multinode split - Delete vllm_singlenode_mi355x.rst; it documented the removed schema and also failed to build (unknown target name at line 113) - Correct the stale suite name and config path in run-cvs-tests.rst, with the test listing taken from live `cvs list vllm` output Ray is documented as opt-in rather than required: mp is the default multinode backend, and ray's effect is to relax the pipeline-parallelism requirement, not to enable multinode. * docs: correct vLLM threshold coverage semantics Threshold files are validated on cell coverage only. The vLLM loader passes an empty gated set, so no metric is mandatory in a cell; a metric is asserted only where its cell carries a spec for it. Gated marks the designated pass/fail criteria that seed the report gate matrix, not a required spec. Point the install guide at the shipped vLLM configs and their current schema keys, replacing a config file that no longer exists.
…aining Suites- (#307) *Added Megatron test suite with Orch refactoring, untouched the legacy Megatron files. This PR introduces a complete Megatron-LM pre-training validation suite for MI325X GPUs, covering single-node and distributed (multi-node) runs. The suite drives Megatron-LM training jobs inside a container, parses training logs, and gates results against configurable per-combo performance and correctness thresholds with a linked HTML report. --------- Signed-off-by: sukesh kalla <skalla@amd.com>
… Training Suites- (#306) *Added Torchtitan test suite with Orch refactoring, untouched the legacy Torchtitan files. This PR introduces a complete Torchtitan pre-training validation suite for MI355X GPUs, covering single-node and distributed (multi-node) runs. The suite drives Torchtitan training jobs inside a container, parses training logs, and gates results against configurable per-combo performance and correctness thresholds with a linked HTML report. --------- Signed-off-by: Rajesh Thummala <rthummal@amd.com>
Signed-off-by: Saravanan Solaiyappan <saravanan.solaiyappan@amd.com>
Signed-off-by: Saravanan Solaiyappan <saravanan.solaiyappan@amd.com>
Two unit tests asserted behavior the implementation no longer has: - ModelSpec now declares precision as an optional field (default ""), so it is accepted, not rejected. Assert it is stored, keep the default-"" check, and add a test that a truly unknown field still fails (_Forbid contract). - BaseVariantConfig.threshold_json is optional (default ""); when absent, discovery falls back to the sibling *threshold.json. Assert it defaults to "" instead of raising. Signed-off-by: Saravanan Solaiyappan <saravanan.solaiyappan@amd.com>
solaiys
requested review from
amd-droy,
anujmittal-amd,
atnair-amd,
cijohnson and
speriaswamy-amd
August 13, 2026 11:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR merges the dev/dtni (DTNI — Deterministic Training & Inference) feature branch back into main, bringing in a large body of inference- and training-suite work developed and
stabilized on dev/dtni over this cycle. The goals:
lifecycle, sweep selector, and reporting engine.
comparison.
dev/dtni was kept linear on top of main via weekly rebases, so this merges as a clean, ordered history. This is the final rebase/merge of the dev/dtni line.
Technical Details
48 commits on top of main (base d4b0223, head 4df77de) — 346 files changed, +62,547 / −5,541. Grouped by area:
Inference reporting engine (new — cvs/lib/report/)
full unit-test coverage under cvs/lib/report/unittests/.
Unified inference suites
OpenAI-compatible smoke test; Prometheus /metrics-derived latency metrics (feat(vllm): add Prometheus /metrics-derived latency metrics (queue/prefill p50/p95) #274); MI300X 14-model workload config set (feat(vllm): add MI300X workload config set (14 models, single + distributed) #298); docs (Document the unified vLLM suite #297).
single/distributed scripts + stale JSONs/docs.
Training suite orchestrator refactors
Core / platform
for pull_image.
Docs & housekeeping
current implementation (adds a _Forbid unknown-field rejection test).
Merge strategy: dev/dtni is 45 rebased commits + 3 follow-up fixes, kept strictly linear on main. Recommend "Rebase and merge" to preserve the linear history and avoid
duplicate/squashed SHAs.
Test Plan
make ut(isolated.test_venvwith pinned deps) runs the fullrun_all_unittests.pysuite covering config loaders, parsers, orchestrator,report engine, and each inference/training lib (mocked orchestrator — no
hardware).
make lint(ruff + pylint logging checks) andmake fmt-check.single‑node on MI300X — per‑sweep rows, thresholds, loss curves, metric‑results
HTML ([JAX][Training] Add JAX MaxText single-node & distributed training suites (sweeps, metrics, thresholds, reporting) #300).
Prometheus metrics, and HTML reports (feat(vllm): unify vllm_single + vllm_distributed into one topology-parametrized suite #257, feat(inference): lm-eval-harness based accuracy evaluation + vllm suite integration #268, SGLANG - orchestration and reporting changes #275, Hnimrama/atom multinode #296).
Test Result
run_all_unittests.pyvia.test_venv);ruff format + lint clean.
producing the expected per‑cell / per‑sweep PASS/FAIL HTML reports.
behavior while consolidating execution and reporting.