Added primus_lib.py

[sukesh-amd]

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[solaiys]

Reviewed for blocking issues + whether it follows the CVS node command-execution convention.

🔴 Blocking — factory.py has a broken import; the dispatcher is un-importable (see inline comment on factory.py). from cvs.lib.training.megatron.megatron_lib import MegatronTrainingJob points at a module that doesn't exist — the real Megatron lib is the flat cvs/lib/megatron_training_lib.py and its class is MegatronLlamaTrainingJob (not MegatronTrainingJob). So importing create_training_job raises ModuleNotFoundError.

Node command-execution convention (the consistency check requested)

• primus_lib.py itself follows the modern CVS orch-based convention used by the current inference suites (vllm/sglang): orch.exec(...) / orch.exec_cmd_list([...]) for in-container execution across nodes, and orch.all.exec(...) / orch.all.exec_cmd_list([...]) for bare-host steps (writing volume-mounted wrapper scripts, date, RDMA/ethtool stat collection), with per-node indexing via range(len(orch.hosts)). The host-writes-script / container-launches-script split is coherent and matches how vllm_job.py drives nodes. ✅
• The inconsistency is in factory.py, not the exec model: it dispatches to a Megatron module/class and an (orch, variant_config, **kwargs) constructor that the existing MegatronLlamaTrainingJob (phdl/docker_lib-based) does not provide — i.e. the factory is written against a future/refactored Megatron that isn't in the tree. Please reconcile both the import path/class name and the constructor contract.

Non-blocking notes

• Dead NaN/Inf check in verify_training_results (primus_lib.py ~L663–670): for result_list in self.training_results_dict[result_key]: for result_val in result_list: — the dict values are list[str], so the inner loop iterates over characters and re.search('nan|inf', <single char>) never matches. Should be a single loop (for result_val in self.training_results_dict[result_key]:). It's redundant with the working poll-time _has_nan_inf_results(), so no false PASS in the common path — but a NaN in an uncovered metric (e.g. grad_norm) would slip through both.
• home_dir = os.path.expanduser("~") is expanded on the runner and then used verbatim in remote host/container paths (log_dir/scripts_dir/data_cache_dir defaults). Fine when homes are homogeneous/NFS-mounted, fragile otherwise.
• Minor: start_training_job(timeout=500) — the timeout parameter is never used.

[solaiys]

🔴 Blocking — this import fails at module load; factory.py (and thus create_training_job) is un-importable. cvs/lib/training/megatron/megatron_lib.py does not exist — there is no cvs/lib/training/megatron/ package. The existing Megatron lib is the flat cvs/lib/megatron_training_lib.py, and its class is named MegatronLlamaTrainingJob, not MegatronTrainingJob. Existing training suites import it as from cvs.lib import megatron_training_lib. As written this raises ModuleNotFoundError (and, even with the path fixed, ImportError on the class name). Fix: from cvs.lib.megatron_training_lib import MegatronLlamaTrainingJob (or add the restructured cvs/lib/training/megatron/megatron_lib.py in this PR) — and reconcile the constructor, since create_training_job calls MegatronTrainingJob(orch, variant_config, **kwargs) but MegatronLlamaTrainingJob uses the older phdl/docker_lib model.